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AESTRACT 

This document reports the procedures and data of a 
study designed to examine the hypotheses that medical practices do 
vary significantly. Detailed data on 76 participants were collected; 
three mechanisms were developed to determine the needs of primary 
care physicians* These mechanism were: subjective analysis by both 
participants and study staff, survey of participants and retesting of 
some participants to determine if their scores improved between the 
beginning and end of the study. Results indicated: (1) while there 
were similarities among individual physician's practices, they do 
vary substantially in terms of identifying individual educational 
needs, (2/ it is possible to identify individual educational needs 
for medical specialists, (3) it is possible to identify individual 
educational needs lor family-practitioners, (4) individual 
educational needs do vary, and (5) it is possible to design personal 
educational programs for physicians based on the health care they are 
called to deliver. (MJM) 
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INTRODUCTION 



A major problem in providing continuing education for practicing physicians 
is identification of their educational needs. Consideration of this problem at the 
University of Wisconsin Department of Postgraduate Medical Education led to the 
thesis that physicians' practices contain many variables. These include the 
patients' problems and the physician's educational and cultural background, ethical 
standards, curiosity, awareness of his deficiencies, milieu of his practice and 
his own unique interests in health and disease. All are determinants of the type 
of practice he builds and quality of care he delivers. 

If medical practices vary, then so must educational needs. Identification of 
these needs and the design of an individual educational program to meet them 
would ensure optimal utilization of the time of both learner and instructor, and 
ultimately benefit the patient. 

Equally, there is some justification for ths assumption that a physician's 
perceived need for continuing education may well be at variance with his real 
need. There are very few established mechanisms by which he can identify 
these real needs. The recent publication of self-evaluatior examinations is an 
attempt to assist physicians in this identification. However, there is no mecha- 
nism in these self-evaluation examinations by which a physician can interpret his 
results in terms of the health care he is called on to deliver. 

To test che thesis that medical practices vary significantly and consequent 
educational needs also vary, a research project was designed to: 




1. Gather data to develop a profile of a physician's practice. 



> 



2. Test the physician in major areas of his practice. 



<J 
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3, Provide educational consultation relevant to his practice profile and 
test results. 

The research was conducted under contracts MIH 70-4008 and N1H 70-4030 
with the liureau of Health Manpower Education, National Institutes of Health, 
between June I, 1968 and June 30, 1971. The first contract (June 1, 1968 - 
December 31, 1969) involved development of the procedures and resources 
required, and experimentation with the cooperation of 37 physicians. The second 
(January 1, 1970 - June 30, 1971) involved changes and improvements of proce- 
dures and reso urces from the first study period, development of new resources, 
and experimentation with the cooperation of 76 physicians. 

This repo ,t will deal primarily with procedures and data from the second 
study phase, ho wever background information and data from the first study phase 
will be presentee, compared, and discussed when it is relevant to the end results. 

DEVELOPMENT OF RESOURCES 

In order to carry out the studies, it was first necessary to develop cer- 
tain resources and methodologies. In phase one, this involved selection of an 
indexing system, development of a medical test bank, and adaptation of a com- 
puter program for test composition and administration. In phase two, it involved 
major changes in the indexing system, a new test bank, development of an 
educational resource index, and new computer programs to store and retrieve 
patient data, test questions, and educational resource information. 

Classification System 

Of the numerous indexing systems investigated, the International Classifica- 
tion of Diseases, Adapted (ICDA), gave the greatest promise of meeting the 
requirements of the study. 

The ICDA has 18 major categories, which are further divided into 111 sub- 
categories, and still further divided Luo 975 sub-sub-categories. In the first 
study phase, the indexing of patient information and test questions was in a 
system based on the 18 major categories. This did not prove sufficiently 
definitive. For the second study phase the patient data and test bank were 
indexed in the most definitive level, the 975 sub- sub-categories. When the 
educational resource index was developed, none of the three indexing levels was 
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suitable. The 18 major categories appeared too broad, and the 111 or 975 too 
specific. A variation, more suited to the apparent needs, was developed. This 
was done by taking the 18 major categories and extracting those diseases"!^ 
conditions most frequently encountered by the 37 physicians in the first study^ 
phase and giving them unique classifications for educational resource indexing. 
This expanded the original 18 categories to 54 and has proved reasonably 
satisfactory* 

By adhering to the ICDA, or adaptations of it, it is possible to make tran- 
sitions from one process in the study to another and compare data at various 
points, since all are based on a common indexing system. 

Test ISank 

It was obvious at the start of the study that a comprehensive test bank, 
covering all areas of medicine encountered by primary care physicians, would be 
required. A major factor in the proposed research would be the ability 1 
identify the educational needs of the physician by testing him within the context 
of the patient problems he was called on to solve. 

In the initial study phase, questions were selected from a number of sources, 
coded in the ICDA, and entered into the test bank. This was done under a high 
level of time pressure, and it proved one of the weaKest links in thit study 
phase. While there were many factors involved in the unsatisfactory quality of 
this initial test bank, the primary problem was that many of the questions were 
not relevant to clinical practice. Consequently this test bank was discarded and 
a new one developed for phase two. 

A major effort was made to establish this clinical relevance, and other 
procedures were instituted to promote the quality of the questions. Appendix 
A gives a detailed discussion of experiments conducted to assure relevance and 
develop efficiency in building the comprehensive resource required, since time 
wa^ again an important factor. 

A cycle was established for processing questions before they would be 
entered into the test bank. A large number of questions were written by physician 
staff members; others were obtained from a variety of sources (departmental 
test files in the medical school, the Ohio and Connecticut Academies of General 
Practice, various self-assessment tests, and the Professional Examining Service 
of New York City). 



As an item was obtained, it was subjected to initial review by a staff mem- 
ber. Initially this was a physician, but experimentation indicated that non- 
medical specialist staff members could make the required judgments equally well. 
These judgments involved whether the question was potentially suitable for die 
purposes of the study, if it was in an acceptable format, and if the wording was 
clear. The questions that survived this review were assigned unique accession 
numbers and sent to practicing physicians. They were asked to make a number 
of judgments on each question (see Exhibit 1) and return them. If a question had 
serious defects in the judgment of the clinician, it was discarded. If it could be 
altered to eliminate the defects, this was done with a staff specialist making 
minor changes or a physician making changes requiring medical background. 
When all pxcJiems pointed out by the clinician were resolved, the question was 
again reviewed for format, grammatical construction, and spelling, and sent to a 
subject area specialist on the medical school faculty. He was asked to make 
judgments as to the scientific accuracy (see Exhibit 2). When all problems 
posed by the subject area specialist were resolved, the question was coded and 
entered into the test bank. The coding involved assignment of one or more 
categories of the 1CDA, whether it was appropriate for general practitioners, 
internists, pediatricians, and surgeons, and a Level of sophistication was assigned. 
These levels were defined as: 

Level 1 - a common clinical situation and "on the spot" decision. 

Level 2 - a decision requiring commonly available diagnostic tests and 
procedures. 

Level 3 - a problem or technique requiring specialized training or diagnostic 
tests to manipulate the information. 

An effort was made to attach a reference citation to each question in the 
test bank. The intent was to make the test a learning situation, so that the 
physician who failed to answer a question correctly would have a convenient 
method of studying that specific area of medicine in greater depth. It also gave 
the clinician an opportunity to confirm the scientific accuracy of the question if 
he challenged it. Brief experiments were conducted with third year medical 
students to determine the most efficient and effective method c. obtaining 
references. ,They found this generally to be through use of the most current 
medical texts available; consulting current journals proved much less efficient. 
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Due to restricted funds and student time, references were not found for all ques- 
tions in the test bank. 

It was also intended that the test bank include a variety of questions 
utilizing visuals. It was thought this would add considerably to the flexibility 
and effectiveness of the testing, since visuals appear essential, or at least 
desirable, in some areas of medical education. A limited number of these items 
were developed, but this was abandoned since the time required to produce such 
a question proved substantially greater and the demands of the study dictated that 
all available time be spent in producing test questions in the most efficient 
manner. 

A test bank of 2,020 questions was developed. Since many of these dealt 
with more than one of the ICDA sub- sub-categories, and were consequently given 
multiple codings, the number available for selection in composition of a single 
test was expanded to 3,755 with appropriate safeguards to assure that the same 
question was not assigned twice under different ICDA codings. (See Appendix B 
for a printout of the test bank). 

Educational Resource Index 

During the first phase of the study, once the physician's individual education- 
al needs had been identified, the staff involvement essentially ended and the 
physician was left to work out his specific educational program as best he could. 
It had been intended that the educational consultant assigned from the medical 
school faculty would be deeply involved in this process, but experience indicated 
that these consultants did not have comprehensive knowledge of educational methods 
and media available and consequently could not be of much assistance. For this 
reason, it was decided to develop an index of available educational resources for 
the second study phase. 

The first step was a letter of inquiry to all major organizations and asso- 
ciations in the country that sponsor continuing education for physicians, asking 
for information on upcoming events and available materials. The Continuing 
Education issue of the Journal of the American Medical Association (JAMA) proved 
a comprehensive source of conferences. In the normal course of its activities, 
the department is informed of educational events and materials as they arc 
scheduled and produced, and a procedure was established whereby this information 
was included in the index as it was received in the office. 
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The intent was to develop as comprehensive a listing as possible, with the 
realization that limited funding and staff time would not permit accumulation of all 
educational resources available in the country. Equally, it was not considered 
possible to make judgments on the quality of the listings. 

As educational resource listings were obtainec, each was coded and entere i 
into the computer, under the following format: 

1. Home study 

a. Audio-visual materials 

b. Programmed instruction 

c. Computer assisted instruction 
tl. Texts and journals 

2. Away study 

a. Conferences 

b. Apprentic e ship s 

Limitations were arbitrarily placed on the inclusion of television video 
tapes and 16 mm films, since the equipment to use these resources is not 
usually readily available or usable for most clinicians. Equally, the texts and 
journals were essentially limited to the Core Content Library developed by 
Norman Steams, M. D. , of Postgraduate Medical Institute, Boston, Mass. 

Once an educational resource was selected, it was coded in an adaptation 
of the ICDA, and in the format given abo^e, and entered into the computer. It 
could now be readily related to the patient data and test results, so that it could 
be retrieved in virtually any combination of that format and the ICDA codings. 
The index is updated each month so that new entries become readily available 
and obsolete ones are deleted. (See Appendix C for a printout of the index). 

Computer Programs 

For the first study phase, it was possible to adapt an existing program of 
the University of Wisconsin Computing Center to meet the anticipated needs of 
the study. This was an on-line, interactive program, with the physician 
receiving his test on a portable teletype in his office directly from the computing 
center via telephone lines. General composition of the test was by hand tabula- 
tion of patient data in the 18 ICDA categories, with selection of specific questions 
done randomly by the computer. 
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Technical problems were of such magnitude tnat this was abandoned for the 
second study phase, w.'th testing to be conducted by the more reliable process of 
paper, pencil, and the U.S. mail>.. This major change, and other factors 
evolving from the first study phase, made it necessary to develop new computer 
programs for the study. 

The first involved storage of both the patient data and the test questions, 
with a sub- routine for retrieval of individualized tests. The second involved 
storage and retrieval of the educational resources. Existing programs, were 
adapted for retrieval of cumulative physician and patient data and cross tabulation 
of data. (See Appendix D for a description of the first program and Appendix E 
for the second). 

OBTAINING PARTICIPANTS 

In April of 1968 the principal investigator presented the goals, and procedures 
of the study at a series of regional continuing education meetings in Wisconsin 
and invited physicians to take part. In addition, a number of telephone contacts 
were made with physicians who had been generally supportive of departmental 
programs in the past. • A goal of 30 participants was set for the first study 
phase. Although difficulty in recruiting participants was anticipated, little persua- 
sion was required. Thirty- seven clinicians took part in that phase; the distribu- 
tion was 2?, in general practice, four in internal medicine, four pediatrics, and 

i 

one surgery. * , 

In the second study phase, the goal was to increase the number, of partici- 
pating physicians to 60. The initial participants were contacted and 35 of the 
original 37 agreed to continue. They were also asked to recommend^ or recruit 
additional participants, which many did. In addition, a number of physicians 
who had heard of the study contacted the department to volunteer. In these ways 
the number grew to 76 for the second study phase. The distribution was 63 
general practitioners, 5 internists, 7 pediatricians, and 1 surgeon. 

No attempt was made in either study phase to recruit a sample representa- 
tive of medical practitioners in Wisconsin. Participants were either pre-selected 
on the basis of past associations, recommended, or volunteers. 
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COLLECTION OF PRACTICE DATA 

To examine the thesis that medical practices do vary significantly, - it was 
necessary to develop methodology 7 by which data could be collected and compared. 
The method adopted was based on categorization of tentative diagnoses in the 
ICDA for every patient contact by the physician during a specific time period. 
• A patient contact was defined as a physician-patient interaction which required 
a medical decision to be made. Such contacts could be by office visit, hospital 
rounds, home visit, or over the telephone. Cumulative data would then give a 
profile of the physician's practice. 

Three methods of collecting patient data were explored. 

1. During the first study phase, a project specialist with experience as a 
medical secretary spent a week in the physician's practice setting and 
received information directly from the physician on each patient contact. 

2. During the second study phase, data was voice recorded into a dictating 
machine by the physician one day a week for four weeks, and then 
transcribed at the central office. 

3. Inj both study phases, a number of participants were asked to make 
^advance predictions of their practice profiles. These were then com- 
pared with actual patient data to determine whether they could define 
their practices with sufficient accuracy to make data collection unneces- 
sary. 

7 

Patient Data ' 

In order to determine a physician's practice profile, it was decided that 
eight items of information would be collected on each patient contact: 

1. Age of patient 

2. Sex of patient 

3. Method or place of contact 

4. Significant presenting signs or symptoms 

5. Major tentative diagnosis 

6. Contributing diagnoses 
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Age, sex, method or place of contact, major T&gnosis, and contrlbii 
diagnosis were all keypunched and entered into the comf*tt£j4 along witii^l 



7. 



8. Treatment and disposition 



Tests ordered 




patient number, physician's code number, and date the contact wasPfriade, The 



from these the physician's practice profPe was constructed. Significant presenting 
signs and symptoms, tests ordered, and treatment and disposition were not 
computerized; this information was used by the project staff if they wished to 
examine in detail the physician's rationale in diagnosis and initial treatment oi 
his patients. 

Physician Data 

Additional information was collected about the physician so that various 
cross tabulations of patient data could be made, n; k j certain judgments made by 
the educational consultant. The form used is attached as Exhibit 3. Important 
items extracted from this form included: 



1. 


Medical specialty 


2. 


Age 


3. 


Years in practice 


4. 


Size of medical community 


5. 


Type of practice 


6. 


Size of patient community 


7. 


Type of community 


8. 


Postgraduate training 



These items were all coded and entered into the computer. 

Visit to the Physician 

A project specialist was recruited with the primary responsibility of 
collecting the physician and patient data. Within general guidelines, she was 
responsible for developing her own methodology and procedures to accomplish this. 
In the first study phare, the visit was of one week's duration; in the second it 



major diagnosis and contributing diagnoses were coded in the ICDA, and it is 
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involved about two hours' time, with reliance on machine dictation of data. Since 
both proved satisfactory for purposes of the study, and the latter is more econom- 
ical, discussion will center on that method. 

Once recruitment of a physician participant was accomplished by a physician 
member of the project staff, it was the responsibility of the specialist to contact 
the physician and make detailed arrangements for his participation. This 
involved selection of a date on which to begin data collection and a visit to the 
physician's office to establish rapport, explain the study, and give specific 
instructions on data collection procedures to the physician ard his secretary or 
nurse. Prior to this visit the specialist completed as much of the Physician 
Information form (Exhibit 3) as possible from such sources as the American 
Medical Association Directory . She would then visit the physician to gather the 
additional data desired, and discuss the data collecting procedures with him and 
his nurse or secretary. In certain instances it also involved assisting the 
physician in the procedure to predict his practice profile in advance for later 
comparison with the patient data. Generally this involved spending an hour with 
the physician and a half~hour with his secretary or nurse. The specialist would 
leave a dictating machine and a supply of tape cassettes with them and confirm 
the dates on which data was to be collected. 

Dictation of Patient Data 

The physician was given some latitude as to the procedure he followed in 
actually dictating the data. It was suggested that he do this after every 3-5 
patients, but some waited until the end of the day and then dictated from the 
charts and records of telephone calls that had accumulated. It was required 
that the data be dictated in four consecutive weeks, one day each week. Normally, 
the physician would record data on Monday of the first week, Tuesday of the 
second week, and so forth. ( One day would be omitted because of the tradition of 
taking a day or afternoon off each week, and data was not collected on weekends. 

The physician was provided with a miniature cassette recorder (Norelco 
Model 85 Pocket Memo) which utilized a 20-minute cassette. 10 minutes per side. 
Normally, one cassette would suffice for a full day's patient data. This procedure 
proved quite satisfactory to both the physician and project staff. While definitive 
data is not available, it apparently took 30-60 minutes each day to record 
patient data. Data was quite complete, since the list of eight items desired was 
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taped to the device as a constant reminder and reference as the physician dic- 
tated. 

Transcribing of Data 

When the physician completed dictation of one day's patient data, he was 
asked to return the cassette to the project office. On receipt, it was transcribed 
by hand onto the Daily Record form (Exhibit 4). This form was developed for 
the first study phase and proved satisfactory to the point that only minor changes 
were required for the second study phase. Experiments were carried out as to 
the efficiency of various types of individuals and the level of medical knowledge 
and training required to transcribe the data. If was found that an intelligent 
college student, trained on the job in medical terminology and spelling, could 
accurately transcribe the data. It was then given to another project specialist 
(a registered nurse) who would assign the ICDA codings to the major and con- 
tributing diagnoses. Initially the nurse's codings were confirmed by a physician 
until this was found to be unnecessary. As the project staff gained experience, 
the nurse was able to delegate coding to the student transcriber, with review of 
the results by the nurse before they were keypunched. 

Development of Practice Profile 

With the data on the physician and his patient contacts entered into the 
computer, information could now be obtained as to his practice profile. This 
would give the physician and project staff information on: 

1. Total number of patient contacts in four days. 

2. Average number of patient contacts per day. 

3. Age ranges of patients. 

4. Proportion of place or method of patient contact. 

5. Distribution of patient contacts within the 975 most definitive 
categories of the ICDA. 

From this latter information, an appropriate test could be developed. 
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TESTING 

In order to best serve as a diagnostic tool, to give the physician and his 
educational consultant some data on which to design an educational program, it 
was thought the test should: 

1. Be directly related to the volumes and distribution of patient contacts 
within the physician's practice profile. 

2. Increase in sophistication as the volume of patients with a specific 
disease or condition increased in the physician's practice profile. 

3. Be sufficiently broad to cover the major part of a physician's practice, 
yet give sufficient depth that judgments could be made in specific 
disease categories, 

4. Be of a reasonable length and time commitment for the practicing 
physician. 

It is important to note that the testing mechanism used primarily measures 
instant recall of factual information, and in some cases the theoretical application 
of that knowledge to a patient care situation. In the view of the project staff, 
it does not purport to measure the quality of medical care delivered by the 
physician participant to his patients. Consequently, the test is considered a use- 
ful diagnostic tool, but not a reliable measurement of actual deficiencies. 

Test Composition 

The procedure by which an individual test would be constructed by the 
computer was based on the criteria listed above. As previously indicated, the 
result was less than satisfactory during the first study phase. Therefore, a 
more complex method was developed for the second phase. 

Both patient data and test questions are indexed in the 18 major categories 
of the ICDA, the 111 sub-categories, and the 975 sub- sub-categories. For 
example, a patient contact with a major diagnosis of diabetes mellitus would be 
coded 3-B-250, with the designations signifying: 

Category 3 - Endocrine, Nutritional & Metabolic 

Sub-category B - Diseases of other Endocrine Glands 

Sub- sub-category 250 - Diabetes Mellitus 
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Each test question dealing with diabetes mellitus would have an identical 
coding, 3-B-250. 

The first step in t^st composition required that the computer go to the one 
(of 18) major ICDA category where the physician had the greatest volume of 
patient contacts and assign questions in any of the 975 sub- sub- categories where 
there were thr?e or more patient contacts. The level of sophistication of the 
questions would increase proportionately to the number of patient contacts in th? 
sub- sub-category. For example, if there were five or more patient contacts the 
computer would assign three questions at level three (the most sophisticated), one 
question at level two, and one at level one. If there were only three patient 
contacts in the sub- sub-category, it would assign no level three questions, one 
level two question, and two level one questions. If there were fewer than three 
patient contacts, no questions would be assigned at that time. A maximum of 
five and minimum of three questions would be assigned in any sub- sub-category 
at this point. 

When the computer had scanned the data and completed the above, it would 
move on to die major ICDA category (of 18) with the second greatest volume of 
patient contacts and repeat the process, and so on through (he 18 major categories 
assigning questions in any sub- sub- category where there were three or more 
patient contacts. At such time as the accumulated test questions reached 125, 
the test would be complete. 

This process was expected to meet all four of the criteria listed on page 
17. Experimentation indicated that a physician could complete a 125-question 
test in about two hour^, thus the limitation placed on the computer as to total 
accumulated questions. 

If the above process was completed and the total assigned questions was 
less than 125, the computer was directed to repeat the process for all sub-sub- 
categories with two patient contacts, and if necessary those with one patient 
contact. In this way, the additional questions were thought to give additional 
supporting test data in these of the 18 major categories constituting the major 
part of the physician's practice. 
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Test Administration 

After the computer had determined test composition, it would print out the 
test items and an answer sheet (see Exhibit 5). The left hi\lf of the answer 
sheet would give the information necessary to score and analyze the test results 
and the right half a form for the physician to use in taking the test. The right 
half was detached, covered with transparent plastic, and the answers covered 
with opaque paint. In taking the test, the physkcian would scratch off the paint 
over what he thought was the correct multiple choice option and would ^e 
immediately informed if he was right or wrong, and if wrong what the correct 
option was. He would return only the answer sheet for scoring. 

Test Analysis 

After the test was scored, the project staff would prepare a "gestalt" 
sheet giving an overview of die practice profile and test results by ICDA category 
(see Exhibit 6). This would be provided to both the physician and his educational 
consultant, along with detailed information on the test results by ICDA sub-sub- 
category. 

It is important to note that the physician is not compared with a pre- 
determined scoring level, or with other physician participants, but rather is in 
competition with himself. The significance of the t^st data is in relative 
performance in the various ICDA categories, with thure where he did less well 
hopefully indicating areas where continuing education effort could bring about 
greatest improvement. 

EDUCATIONAL CONSULTATION 

The concept of educational consultation was not clearly defined at the 
beginning of the study, and it was anticipated that the mechanics would evolve 
during the study. The one preconception was that the practicing physician would 
require some assistance in analyzing his own data and developing a meaningful 
continuing education program. 

In the first study phase consultation was provided by calling on members of 
the medical school faculty. An attempt was made to match the consultant's field 
of expertise to the practice profile of the clinician. The plan was to have the 
consultant visit the physician in his practice setting to discuss the data and 
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design a continuing education program, and then establish a continuing relation- 
ship which would provide the clinician an entry point into academic medicine so 
that he could discover the various ways in which his educational needs could be 
met. This was successful in that it did establish the desired rapport in many 
cases. It also provided insights into the clinical practice nf medicine for the 
faculty member. However, the faculty consultants found themselves ill-equipped 
to recommend specific courses and materials for study; consequently the con- 
sultation did not appear to be greatly beneficial to the participant except in 
limited cases. 

For the second study phase, it was decided to limit the consultation role to 
a few faculty members who were fully acquainted with the goals and administrative 
details of the study, and to provide them and the physician with the index of 
available educational resources. 

As it evolved, in both phases of the study, the objective datr of the practice 
profile and test results were used along with the intuitive judgments of the con- 
sultant and practitioner to arrive at and carry out the educational program. 
This was an informal, cooperative relationship between the two. 

The procedure was initiated when the practice and test information was com- 
plete. Both the consultant and physician were provided advance copies of the 
data. The key item of the report was the "gestait" sheet which gave an easily 
observable overview of the profile and test results. Both also received a general 
analysis of the data. In addMon, the consultant had detailed information of test 
results by 1CDA sub- sub- category' and a copy of the test so that he could examine 
this in great detail if desired. 

The consultant would then spend about two hours visiting the physician in 
his practice setting. The visit included both a detailed discussion of the data and 
an informal discussion of the physician's practice and his views of that practice. 
Such questions as: "What part of your practice do you enjoy the most?' 1 and 
"What part do you enjoy the least?" often added meaningful information to the 
objective data provided. The consultant and physician would then cooperatively 
develop a personalized plan of continuing education for the physician and examine 
the educational resource index to determine if there were suitable events or 
materials available. 
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Upon return to the office, the consultant would provide a report of the visit 
(see Exhibit 7) to the project staff and a final report would be prepared for the 
physician (see Exhibit 8). This included printouts of the appropriate sections of 
the educational resource index, with those items thought to Lc most appropriate 
indicated. The consultant would then send the final report to the participant 
along with a personal letter summarizing the recommended educational program. 

EVALUATION 

9 

In a sense, many of the procedures in the study provide a sort of evaluation, 
related to process analysis. This involves the question of whether the study 
staff did in fact carry out those procedures which they set out to do, and did 
they effectively explore various alternatives of accomplishing the same thing more 
effectively and efficiently. The answers to the various facets of this question 
will be provided in the body of the report. 

A second, and more important, question is whether the process did in fact 
provide a mechanism to identify and meet individual educational needs of primary 
care physicians. Three mechanisms were used in an attempt to determine this: 

1 # Subjective analysis by both the participants and study staff. 

2. Survey of participants. 

3. Re-testing of some participants to determine if their scores improved 
between the beginning and end of the study. 

All three methods were used at the conclusion of die first study phase. 
The subjective analysis was obtained by holding a day-long meeting of all 
participants, consultants, staff and representatives of the Bureau of Health Man- 
power Education. The input of the participants was such that it led to major 
changes in the procedures during the second study phase. 

The survey of participants also rendered encouraging data. 

ITie re-testing was limited to six of the 37 participants in the first phase, 
since the inadequacy of the test bank led to uncertainty as to the validity of the 
results. The test scores of those six, however, did illustrate improvement 
except for one participant who did not carry out his educational program, and he 
showed a decrease. 
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In the second study phase the participants were surveyed, (see Exhibit 9) 
but not re-tested. It has not been possible to hold a meeting of all participants 
because of shortage of funds. It is still the hope of the study staff to hold such 
a meeting, but it is not anticipated prior to submission of this report, . 

A third type of evaluation, in a sense, relates to one of the goals of the 
second study phase - development of a reasonable program which could be offered 
to a significant number of physicians on a fee basis. When this is offered, the 
willingness of physicians to pay for such a service, and their satisfaction on com- 
pletion of the process, will be still another measure of the success or failure of 
the research project. 
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RESULTS 



Data Collection 

In general, the data collection efforts during both phases of the study not 
only rendered most of the information required to meet the objectives of the study, 
brt resulted in accumulation of data which has broad application beyond the limits 
of this research. 

Physician Data 

As previously described, considerable data were gathered on each participant 
in the study, cither from directories or the physician himself. The information 
collected was of use both to the consultant (to give him a more complete picture 
of the participant and his practice setting), and in analysis of the data. 

Detailed data on the 76 participants is attached as Exhibit 10. Summary 
information is as follows: 

Table 1 
Medical Specialty 

Number 



General Practice 63 

Pediatrics 7 

Internal Medicine 5 

Surgery 1 

Total 76 
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Table 2 
Age of Participants 











Years 












26 


31 


36 


41 


46 


51 


56 


61 






to 


to 


to 


to 


to 


I 


to 


to 


Average 




30 


35 


40 


45 


50 


55 


60 


65 


Age 


General Practice 


2 


12 


16 


18 


9 


2 


2 


2 


44.0 


Pediatrics 




2 




2 


2 


1 






43.0 


Internal Medicine 




2 


1 




2 








40.2 


Surgery 




1 














34.0 


Total 


2 


17 


17 


20 


13 


3 


2 


2 


41. 5 


(Note: Data were 


also col 


lected 


on " 


Years in 


Practice" 


but the 


correlation is so 



close with physician age that the data are not reported here). 



Table 3 

Type of Practice 
(by number of participants) 

| Group ■ 1 

Uni- Uni- Multi- Multi- 

Specialty Specialty Specialty Specialty 

Solo Under 5 Over 5 Under 5 Over 5 

General Practice 8 22 5 4 24 

Pediatrics 7 

Internal Medicine 5 

Surgery 1 

Total 8 22 5 4 37 
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Table 4 



Size of Patient Community 
(by number of participants) 

Population in 1,000s 





Less 


2 


5 


10 


24 


50 


100 






tli an 


to 


to 


to 


to 


to 


to 


Over 




2.5 


5.4 


9.9 


24 


49 


99 


499 


500 


General Practice 


13 


15 


15 


5 


6 


0 


8 


1 


Pediatrics 






3 




I 


3 






Internal Medicine 










1 


4 






Surgery 








1 










Total 


13 


15 


18 


e • 


8 


7 


e 


1 



General Practice 
Pediatrics 
Internal Medicine 
Surgery 

Total 



Table 5 

Office Setting 
{by number of ' participants) 



Business 
District 

24 

2 



26 



Residential 
26 
5 
3 
1 



35 



Suburban 
Business 

10 

2 



12 



Rural - 
Residential 



General Practice 
Pediatrics 
Internal Medicine 
Surgery 

Total 



_0_ 
41 
1 



42 



Table 6 
Postgraduate Training 



17 



17 



_2 
5 
4 



Years 
3 



2 
2 



2 

_i_ 

3 



0 
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As previously indicated, no effort was made to obtain a study sample repre- 
sentative of Wisconsin 'physicians. Various comparisons of the study group with 
general Wisconsin data are as follows: 



Table 7 

Age Distribution 
(by per cent) 

(Participants compared with Wisconsin direct care physicians) 

Years 





26 


31 


36 


41 


46 


51 


56 


61 








to 


to 


to 


to 


to 


to 


to 


to 




Ave. 




30 


35 


40 


45 


50 


55 


60 


65 


65 + 


Age 


Participants 


2.6 


22.4 


22.4 


26.3 


17. 1 


3.9 


2.6 


2.6 


00.0 


41. 5 


Wisconsin M. D. s 


1.8 


9.7 


16, 5 


16.4 


16.4 


10.8 


9.5 


8.8 


10.2 


46.0 



Table 8 



Distribution by Specialty Practic e 
(by per cent) 

(Participants compared with Wisconsin physicians in same specialties) 



Participants 
Wisconsin M. D. s 



General 
Practice 

82.9 
53.4 



Pediatrics 
9.2 
8.3 



Internal 
Medicine 

6.6 

20.2 



General 
Surgery 

1.3 

18. 1 



Participants 
Wisconsin M. D. s 



Table 9 

Distribution by Type of Practice 
(by per cent) 



Solo 
10.5 
40.2 



Group 
89*5 
45.2 



Other 
0.0 
14.6 



Consequently, the 76 physicians participating in the siudy, when compared 
with representative samples, ar i * younger by an average of 4.5 years, include a 
disproportionately large number of general practitioners, and tend more to group 
practice. 
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No comparative data were available to niako similar determinations concern- 
ing the more definitive listing of large and snvll group practitioners, unispecialty 
or multi-specialty group practice, size of patient population, office setting, or 
number of years of postgraduate training. 

In summary, the participants average 41.5 years of age, with two-thirds 
between 31 and 45 years of age. They are all primary care physicians, but are 
mostly in general practice, with limited representation in pediatrics and internal 
medicine and one surgeon. 

All of the specialists and most of the general practitioners in the study are 
in group practice, with only eight solo practitioners involved. Those in groups 
are almost evenly divided between small and large groups and unispecialty (all 
general practitioners, all pediatricians, etc.), and multi-specialty groups. 

Approximately one-third have their offices in central business districts and 
one-third in residential areas. Ten have offices in suburban business districts 
and three in residential areas developing on the outskirts of communities. 

More than half the participants did not have any formal postgraduate training 
beyond internship; the remainder took residency training of one to six years. Of 
interest are the 22 general practitioners who had one or two years of residency 
training and 1 pediatrician who had none. 

The data reported .here about the physician participants has interest and 
vrlue to the project staff, particularly to the educational consultant, and in cross - 
tabulation of cumulative data. It has little interest to the individual practitioner, 
and was not utilized to any great extent in planning personalized educational pro- 
grams. However, there is merit in its collection since the cumulative informa- 
tion has application beyond the limits of the study and in this context it makes 
the patient data and test results more meaningful. 

Patient Data 

As previously indicated, two methods of data collection were utilized during 
the study. Also, physicians were asked to predict their profiles to determine if 
the data collection phase could be dispensed with. 

There was apprehension that the change from having a medical secretary 
collect data in phase one to dictation of data by the physician in phase two would 



22 



result in loss of data. It would appear that this occurred to a limited extent, if 
at all. 

This conclusion was drawn after examining the data and contacting a number 
of physicians. Comparison of volume of patient contacts for the two study periods 
is: 



Table 10 





Average Daily 


Volume of 


Patient Contacts 








Office 


Phone 


Hospital 


Home 


Total 


First Study Period 


28.7 


15.7 


9.3 


0.4 


53.9 


Second Study Period 


28.0 


8.9 


9.8 


0.5 


48.2 


Difference 


- .7 


- 8.9 


+ . 5 


+ . 1 


- 8.7 



The decrease in total average daily patient contacts between the two study 
periods is almost identical with the clecrease in telephone contacts. The trend 
was detected early in the second study phase, and individual physicians contacted 
to determine if they were neglecting to record all of their telephone contacts. 
This revealed that a major fartor was a change in practice procedure by some 
of the physicians who participated in the first phase; they were now having their 
nurse receive telephone calls, handle those within the scope of her training, and 
pass on only those requiring die attention of the physician. There is still a feel- 
ing that some of the decrea.se is due to missing data, but there is no way to 
document th s. 

Since the results under the two methods correlate closely, the relative cost 
becomes a determining factor. Identifiable costs in travel and supporting a medi- 
cal secretary in the community for a week resulted in an average of $235 in 
direct costs of obtaining data (or one participant during the first study phase. 
This decreases considerably when multiple members of a group are profiled con- 
currently. In the second method, a staff member did visit the physician in his 
community, but only on one day. Therefore, the cost of this visit and transcrib- 
ing of the data in the office came to $44.25 in identifiable costs for each partici- 
pant in the second study phase. These costs decrease also when multiple mem- 
bers of a group can be contacted in a single day, but the cost of transcription 
remains constant per physician. The second method also is thought to give more 
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representative data, since it permits sampling by recording data one day a week 
for a niontn, rather than lour days in a single week. 

The conclusion, primarily on the basis of cost, is that the method of choice 
in gathering patient data is by dictating machine, and that this can be done with 
relatively little loss of data. It is entirely possible that the visit of the staff 
member to orient the physician and his secretary may be dispensed with; this is 
a probability with physicians who have previously participated in the project. 

Predic t :> n of Profile 

In an attempt to determine if physicians could predict their practice profiles 
with the degree of accuracy required for the procedure, experiments were con- 
ducted during both study phases. 

During phase l f 17 partici pants were asked to record in advance the per- 
centages of their practices they thought would fall into each of the 18 1CDA 
categories. 

During the second phase, 50 participants were asked to select the 6-8 ICDA 
categories in which they thought they would see the most patients, and then 
specify the numbers of patients they, would predict under the sub-sub-categories 
in those major categories selected. 

The results were as recorded in Table II on the following page. 

In both phases the majority of tne participants were able to predict between 
62.5 and 75.0 per cent of the categories which constituted the major portions of 
their practices. 

It had been anticipated that those physicians who participated in phase 1 
would be more successful in predicting their profiles in phase 2 than the new 
participants. This did not prove to be the case. 

Effect of Utilizing Predicted Profiles 

Any judgment based on the data in Table 11, whether approximately 70% 
accuracy in predicting practice profiles is acceptable, would be subjective. A 
more objective approach would be to consider the effect on test composition, since 
this is the major use of patient data in the study. 
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Table 11 

Prediction of Practice Profiles 
(by number of participants) 

Categories Phase Two 

Per cent Predicted Repeat New 

Accuracy Accurately Phase One Participants Participants 

25.0% 2 of 8 1 

28.6% 2 of 7 1 1 

42.9% 3 of 7 1 

50.0% 4 of 8 1 2 .1 

55. 5% 5 of 9 1 

57. 1% 4 of 7 2 

60.0% 6 of 10 i 



62. 5% 


5 of 8 


4 


4 


6 


71.4% 


5 of 7 




3 


2 


75. 0% 


6 of 8 


10 


7 


9 


83.3% 


5 of 6 




1 




85.7% 


6 of 7 






1 


87. 5% 


7 of 8 


1 


3 


3 


100. 0% 


8 of 8 


1 










17 


21 


29 


Average Accuracy 


72. 8% 


69. 7% 


66. 1% 



During phase 1, utilizing the data collected by the medical secretary on the 
17 physicians who also predicted their profiles, the best result was one physician 
who was tested on 98. 2 per cent of his practice; the worst result was one who was 
tested on 71.5 per cent. Had the tests been composed on the predicted profiles, 
the best result would have been 78.6 per cent and the worst 46.0. Consequently, 
the loss in diagnostic effectiveness of the testing procedure would have ranged be- 
tween 19.6 and 25.5 per cent for the 17 physicians involved. 

At the end of phase 1, an analysis of the data on predicted profiles led to 
the conclusion that it was not sufficiently accurate or consistent to provide a sub- 
stitute for recording patient data. At that time a 100 question test was adminis- 
tered, and the net effect would have been to reduce the number of questions 
related to actual patient contacts to a range of 46-79. There was already doubt 
that 100 relevant questions represented an adequate test; the lower number 
appeared unacceptable. 
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The results during phase 2 were even less encouraging. The percentages of 
actual practices which would have been covered by tests based on predicted pro- 
files ranged from 80.0 to 28.0 per cent. The actual tests, based on the data 
collected by dictating machine, covered between 94.5 and 80.0 of their practices. 
The loss in diagnostic effectiveness would have Tanged between 14.5 and 56.0 per 
cent. 

More important, in phase 2 the test formula assured that all 125 questions 
on the test related to actual patient contacts. If the predicted profiles were 
utilized in test composition this would have been reduced to 100 in the best in- 
stance and 35 in the worst. 

Conclusion 

On the basis of this data, predicting of practice profiles would be an accept- 
able substitute for actual recording of data only if cost became die crucial factor. 
One might accept the reduced effectiveness if the only other alternative were to 
discontinue the entire program because of the cost involved. 

There is some evidence that physicians see value in the process conducted 
on the basis of profile prediction. This was attempted hi an issue of Patient 
Care magazine. A description of the process was given and the reader allowed 
to edict his profile and then take one of three standardized tests. The test 
results were mailed in and the appropriate portion of the Educational Resource 
Index mailed back to the physician. A total of 143 physicians took part, but 
there are no data available to indicate how useful this was to them. The experi- 
ment was conducted and fincinced outside the basic contract, and is reported here 
as a matter of interest. 

Limitations of Patient Data 

There are limitations to the methods used in collecting patient data. The 
first is that the most important item is the major tentative diagnosis. This 
assumes that this initial diagnosis is correct, that the physician does not change- 
his diagnosis as the result of later evidence, or that he has not completely mis- 
diagnosed the case. While it is an admitted weakness, it is not considered 
feasible to follow the progress of patients and change the data if the diagnosis 
changes, or to have another physician "second guess" the participant on his 
initial diagnosis, when he has not actually had contact with the patient. 
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The second problem arises from the fact that only sampling of patient data 
is done. In phase 1 this was lour days out of a week; in phase 2 four days out 
of a month. Some participants have indicated they feel a greater volume of data 
over a greater time span would more nccurately reflect their practices. It be- 
comes a question of what is acceptable in terms of cost and demands on the 
physician's time, as opposed to the ideal. This may be partially alleviated for 
the physician who adopts Individual Physician Profile as a continuous process, and 
participates over a number of years. In this way, by selection of data collection 
periods, seasonal changes in practice could be detected and stable areas of 
practice confirmed. 

General Patient Data 

From the data collected, it was possible to furnish ea^h participant with 
general information on his practice: 

1. Age ranges of his patients 

2. Sex distribution of his patients 

3. Method or place of contact 

4. Distribution of patients by diagnosis 

During this study it was not possible to give the participant comparative 
data so that he could analyze his practice in relation to others. With the cumula- 
tive data of phase 2, it will be possible to do this in the future. The cumulative 
data is as follows: 

Table 12 





Age 


Range 


s of Patients 










(Average 


Patient 


Contacts Per 


Day) 










0 


15 


40 










to 


to 


to 


Over 


Not 




Total 


14 


39 


64 


65 


Reported 


General Practice 


530 


12 


16 


11 


8 


3 


Pediatrics 


46 


38 


6 


0.4 


0. 1 


2 


Internal Medicine 


31 


0 


6 


10 


8.4 


6 


Surgery 


37 


_4 


_5 


14 


10 


4 


Total Average 


48.2 


14 


14 


10 


7 


3 



EMC 



27 



Table 13 

Sex of Patients 
(Average Patient Contacts Per Day and Per cent) 

• Not 





Male 


% 


Female 


% 


Reported 


% 


General Practice 


19 


38.6 


29 


57.1 


2 


4.2 


Pediatrics 


20 


43.0 


20 


43.0 


7 


15.0 


Internal Medicine 


13 


' 43.0 


16 


51.0 


2 


5.8 


Surgery 


n 


30.0 


24 


66.0 


1_ 


4.0 


Total Average 


18 


39. 1 


27 


55.6 


5 


5.3 



Table 14 

Method or Place of Contact 
(Average Patient Contacts Per Day and Per cent) 

General Practice 
Pediatrics 
Internal Medicine 
Surgery 

Total Average 

Of this general data, the most useful to the participant has invariably been 
the breakdown of the method or place of patient contact, and particularly the 
volume of medicine practiced over the telephone. In many instances it has re- 
suited in significant changes ir office procedure, and each participant is now 
given a reference where he may obtain an office guide for telephone answering 
procedures. 

As with most research, there is a great deal of serendipity involved in the 
project - a significant amount of valuable but unanticipated information has evolved 
from die 'data. However, since it is not directly related to the main objective of 
diagnosing and meeting individual educational needs, it will be presented later in 
this report. 

Patient Data by Diagnoses 

Ail of the patient data reported here deals with the second study phase, 
except in instances where comparison with data from the first study phase illus- 
trates a particular point. 




Office % 

30 60.4 

26 56.0 

16 5i.O 

_5 14.0 

29 59. 5 



Phone % 

9 17.7 

12 26.0 

6 18.0 

_8 22.0 

9 18.4 



Hosp. % 

10 19.7 

8 16.0 

9 29.4 
20 56.3 

9 19.3 



Home % 

0.6 1.1 
0.0 0 
0.0 0 
0.0 0 

0.5 1.0 



The 76 physicians in the second study phase provided health care for 14,507 
patients during the 301 days of practice on which data were collected. Three 
participants collected data on only three days because of unanticipated problems; 
the remaining 73 collected four days 1 practice data. A total of 14,486 primary 
diagnoses were made; there were no diagnoses for 21 patients. 

Also included in the data are from one to five secondary diagnoses for cer- 
tain patients. These evolved in two ways. The first is when a patient has more 
than one immediate problem or reason for contacting the physician; the less 
serious were tabulated as secondary diagnoses. In another instance, the patient 
may have a prior disease or condition, not involved in the reason for contacting 
the physician, but having implications in the diagnosis and treatment. An example 
would be a diabetic patient, contacting the physician with a cardiac problem. The 
diabetes may be well controlled, and not a factor in the patient's reason for the 
visit, but it is important in the physician's treatment of the cardiac problem. 
Therefore it is included in his practice profile. By including these secondary 
diagnoses, the number on which the practice profiles are based increases to 
23,911. The distribution of these is given in Exhibit 11. 

Analysis of Individual Practices 

On the assumption that die data collection procedure does accurately record 
each physician's practice, the question then arises as to the similarities and 
differences between practice profiles of the 76 participants. 

In the final analysis, the procedure is equally valid whether or not there 
are correlations between practices. However, it would be greatly simplified and 
less costly if correlations were sufficiently high to establish "standard" practice 
profiles for physicians meeting certain criteria. If one could accomplish the 
same objectives in personalized planning of continuing education at a lower cost 
and lesser demand on the physician's time, it would make broad application of 
the research results much easier. 

One of the major theses of the study is that physicians have varying 
practices and consequently unique educational needs within the context of their 
personal scientific knowledge in relation to the health care they are called on to 
deliver. Analysis of the patient data appears to both support and contradict that 
thesis. 
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Contradictory Evidence 

The major contradiction lies in the fact that five of the 18 ICDA categories 
appear to predominate in the majority cf the practice profiles. These five arc, 
in order of frequency: 

18 - Special Conditions & Examinations without Illness 

7 - Diseases of the Circulatory System 

8 - Diseases of the Respiratory System 

16 - Symptoms and Ill-defined Conditions 

17 - Fractures, Trauma and Poisoning 

In all, these five categories account for 63.14 percent of the 23,911 diagnoses 
recorded. Next in rank order of volumes of patient contacts are categories with 
approximately equal numbers: 

< 9 - Diseases of the Digestive System 
10 - Diseases of the Genitourinary System 
3 - Endocrine, Nutritional & Metabolic Diseases 

These three disease categories account for an additional 14.34 percent of 
total diagnoses, bringing die total of eight categories to 77.48 percent. Next in 
order are two categories with approximately equal volumes: 

6 - Diseases of the Nervous -System & Sense Organs 
5 - Mental Disorders 

These account for 7,12 percent of total diagnoses, bringing the total of the 
10 categories to 84.58 percent. Next in order are four categories with approxi- 
mately equal volumes: 

12 - Diseases of the Skin & Subcutaneous Tissue 

13 - Diseases of the Musculoskeletal System & Connective Tissue 

1 - Infective & Parasitic Diseases 

2 - Neoplasms 

These four account fo>- 12.19 percent of total diagnoses, beinging the total 
for 14 categories to 96. 77 percent. 

With this distribution, one would anticipate certain correlations to occur 
among the practice profiles of the 76 participants. Some general correlations by 
ICDA category follow based on rank ordering the 18 major ICDA categories in 



30 



each profile and determining the frequency with which rank orders correlate, 
depending on the number of categories compared. 



Table 15 

Rank Order Correlations by 1CDA Categories 



Main Category 



Category Frequency 



18 


CO 

bZ 


7 


1 0 

IZ 


8 


c 
D 


16 


c: 
0 


17 


Z 


All others* 


u 




IE 


Three Categories 




Categories 


Frequency 


7, 16, 18 


9 


8, 16, 18 


9 


8, 17, 18 


8 


7, 17, 18 


7 


7, 8, 18 


7 


16, 17, 18 


4 


All others* 
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IE 



Two Categories 



riarepories 




Frequency 


8, 18 




24 


7, 18 




16 


16, 2 8 




14 


17, 18 




11 


7, 16 




6 


8, 17 




2 


All others* 


Four Categories 


3 
76" 


Categories 




Frequency 


7, 8, 16, 


18 


8 


8, 16, 17, 


18 


7 


1, 8, 16, 


18 


2 


1, 8, 17, 


18 


2 


2, 7, 16, 


18 


2 


All others* 




55 
76 



* (None has a frequency greater than one) 



Five Categories 



Six Categories 



Categories 




Frequency 


Categories 








Frequency 


7, 8, 16, 17, 


18 


23 


6, 7, 


8, 


16, 


17, 


18 


5 


3, 7, 8, 17, 


18 


3 


7, 8, 


10, 


16, 


17, 


18 


5 


3, 7, 8, 16, 


18 


2 


7, 8, 


9, 


16, 


17, 


18 


3 


7, 9, 16, 17, 


18 


? 


1. 7, 


8, 


16, 


17, 


18 


2 


All others* 




46 


2, 7, 


8, 


16. 


17 


18 


2 






75" 


3, 7, 


8, 


16, 


17, 


18 ' 


2 








5, 7, 


8, 


16, 


17, 


18 


2 








12, 7, 


8, 


16, 


17, 


18 


2 








13, 7, 


8, 


16, 


17, 


18 


2 








All others* 








51 
77 



Seven Categories 

Categories 

3, 7, 8, 10, 16, 17, 18 

2, 5, 7, 8, 16, 17, 18 

2, 7, 8 f 9, 16, 17, 18 

3, 5 t 7, 8, 16, 17, 18 
All others* 



Frequency 

3 
2 

2 , 
2 
67 



Beyond this point of analysis, all 76 practices become unique; there is no 
combination of eight categories with a frequency greater than one. 

As would be anticipated, there is high correlation when only the major 
category with the greatest volume of diagnoses in each profile is considered; this 
is category 18 for 52 of the 76 participants. As one increases the number o! 
ICDA categories which must correlate, the frequency decreases up to the point 
that five categories are considered- Here 23 of the 76 show a correlation. 

Examining additional; data available on these 23 physicians 1 practices reveals 
that there are common criteria: (1) all are general practitioners, (2) 19 (82.61%) 



(None has a frequency greater than one) 
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are in solo or small group practice, and (3) 20 (86.96%) practice in communities 
of less than 10,000 population. More important, 17 of the 23 (73.91%) meet all 
three criteria. 

Therefore, in an analysis on this basis, it would appear that the thesis 
that all medical practices vary is only partially true; 36.51 of the general prac- 
tices in the study appear to have a degree of correlation. 

Supporting Data 

Carrying the analysis one step further, by applying the three criteria above 
to the remaining 53 participants, one finds 13 general practitioners in solo or 
small group practice, in communities of less than 10,000 population, whose 
practice profiles do not fit this pattern. Therefore, while a practice pattern 
does exist for more than one-third of the general practitioners in the study, there 
appears to be no way to identify in advance which ones might fit this pattern. 

The results are sufficiently promising, however, to warrant further analysis. 
Since the diagnoses in the practice profile are used to determine test composition, 
perhaps this is the most valid criterion to use in comparison. To this point 
correlation has been attempted only in the 18 broad ICDA categories. Since the 
test is composed at the level of the 975 sub-sub-categories, there must also ba 
correlation at this level. 

With this in mind, a more detailed analysis was done of the practices of the 
eight solo practitioners who met the three criteria previously stated. Four were 
among the 23 showing a degree of correlation; four were not. An attempt was 
made to develop a standard profile and devise a standard test for these eight 
practitioners. Their practices were compared at the level of 975 sub-sub-cate- 
gories, and the profile constructed to include those diseases and conditions where 
there were the greatest volumes of diagnoses, and the broadest distribution. 
While the aim was that each of the eight would have at least one diagnosis in any 
area where questions were assigned, this was not always possible. By trial and 
error, it was determined that the best standard profile, covering the greatest 
amount of the eight practices occurred when 10 oi the major ICDA categories 
were included. 
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The standard profile which evolves, and assignment of test items, is as 



follows: 

Number of 

Category 3 (Endocrine, Nutritional and Metabolic) Questions 

B-250 Diabetes inellitus 5 , 

B-277 Obesity not of endocrine origin 5 

Category 5 (Mental Disorders) 

B-300 Neuroses 5 

*B-305 Physical disorders of psychogenic origin 5 

Category 7 (Circulatory System) 

C-401 Essential benign hypertension 5 

D-412 Chronic ischemic heart disease 5 

G-440 Arteriosclerosis 5 

Category 8 (Respiratory System) 

*A-463 Acute tonsillitis 5 

A-465 Acute upper respiratory infection 5 

D-493 Asthma 5 

Category 9 (Digestive System) 

*D-551 Hernia of abdominal cavity without obstruction 5 

*F-574 Cholelithiasis 5 

Category 10 (Genitourinary System) 

B-599 Other diseases of urinary tract 5 

*E-627 Menopausal symptoms 5 

Category 12 (Skin and Subcutaneous Tissue) 

*A-682 Other cellulitis and abcess 5 

* B-692 Other eczema and dermatitis 5 

Category 16 (Symptoms and 111 -defined Conditions) 

*A-784 Symptoms referable to upper respiratory tract 5 

*A-786 Symptoms referable to genitourinary system 5 

*A-787 Symptoms referable to limbs and joints 5 
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Category 17 (Accidents, Poisonings and Violence) 

*E-845 Sprains and strains of ankle and foot 

* 1-882 Open wound of hand 

* J -891 Open wound of knee, leg and ankle 



Number of 
Questions 



5 



5 



5 



Category 18 (Conditions and Examinations without Illness) 



A -000 General medical examination 



5 



C-020 Prophylactic inoculation and vaccination 
K-105 Surgical aftercare 



5 



5 

125 



Those diseases and conditions marked (*) in the standard profile indicate 
occasions where from one to three of the eight physicians would receive test 
questions when they did not report diagnoses in those areas. A detailed analysis 
shows that one physician will receive 20 questions not related to diagnoses he 
reported, three will receive 15 questions, one 10 questions, two five questions, 
and one will have all questions related to reported diagnoses. 

Some arbitrary judgments have been made in devising this standard test. 
The consultants indicate that if less than five questions are asked on o specific 
disease or condition, it is difficult to make a judgment on the results. There- 
fore, this was set as the minimum. It was also made the maximum, so that the 
examination could cover as many disease categories as possible. A limit of 125 
questions was set, si r :e experience indicates it will take two hours for adminis- 
tration, and this is thought to be the maximum feasible for a busy physician. 

The following tabulation compares the comprehensiveness of the examination 
actually administered to the eight physicians with the comprehensiveness of the 
standard test. The comparison is made at the level of 975 sub-sub-categories; 
the percentages indicate the number of diagnoses on which questions were asked 
in the actual test, compared to total diagnoses, and the number of diagnoses on 
which questions would be asked in the standard test, related to total diagnoses. 




Table 16 

Percent of Practice Covered by Test 
Total Profile 10 Major Categories 





Actual 


Standard 


% of 


Actual 


Standard 


Code Number 


Test 


Test 


Practice 


Test 


Test 


Correlated Profiles 










16049 


64.1 


zo. 1 


oo. 1 


11. 1 


OO. / 


21455 


54.8 


9Q Q 
lo . 0 


OA A 

84. 4 


An 1 
OU. o 




46540 


52.8 


OC Q 


QQ 1 

OO, 1 


CC Q 

Do. o 


O Z 4 O 


52030 


59.3 


OA 1 


"7Q A 

/o. o 


OO. o 


^o. U 


Average 


57.8 


9ft n 


ft 1 1 


A3 n 

OO • u 




Non- correlated 


Profiles 










36151 


56.1 


29. 0 4 


82.4 


63.8 


39.5 


51328 


53.9 


29.8 


81.2 


70.2 


37. 1 


62392 


53.5 


39.2 


81.9 


63.4 


47.0 


63053 


61.9 


26.6 


83.3 


63.1 


32.8 


Average 


56.4 


31.2 


81.2 


65. 1 


39.1 


Overall Ave. 


57.1 


29.6 


SI. 2 


64.0 


36.8 



The percentages listed above are lower than those listed for similar 
comparisons elsewhere in this report, since the criteria are much more stringent; 
in previous cases correlation was at the level of 18 categories and here it is at 
the level of the 975 sub-sub-catcgories. 

In any event, use of the patient data in designing a test .appears much more 
effective when practice profiles are considered individually, than when they are 
correlated tp produce a standard test. Four of the eight profiles analyzed are 
among those that would be expected to be most appropriate for development of a 
standard profile; four are not. There appears to be little difference among the 
eight in the results of simulating this. 

Consequently, without conducting similar analyses of the other 68 profiles in 
the study, it appears reasonable to conclude that a great deal of the diagnostic 
value of the test would be lost in the attempt to simplify the procedure by 
constructing standard profiles and standard tests. 
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Additional Observations 

Up to this point, only those portions of the data having a direct relationship 
to the central purposes of the study have been discussed. The physician and 
patient data also renders some interesting information concerning medical practice 
which may have application outside the parameters of the study. 

For example, the cumulative data confirms previous studies in which it is 
found that approximately 80% of medical practice is in the office and 20% in the 
hospital. (The most recent statistics, based on "A Study of General Practice in 
iMassachusetts", by Brown, et. ah, in the April 12, 1971 issue of JAMA , gives 
the figures as 71.6% office, 21,6% hospital, 5.6% home visits, and 1.2% other.) 

The data involving general factors in physicians 1 practices also reveals 
numerous other interesting items when cross -tabulated with various factors. For 
example, the volume of patients when compared with the age of the physician: 

Table 17 

Average Daily Patient Contacts by Physician Age 

26 31 36 41 46 51 56 61 
to to to to to to to to 
^35404550556065 

Number of Physicians 3 16 17 20 13 3 2 2 

Ave. Daily Contacts 22 40 54 55 47 43 55 45 

It would appear that the physician entering practice builds his practice 
volume up to age 45, reaching n maximum workload at age 36-45, and then 
decreases somewhat and maintains that level until retirement. The only variation 
in the data occurs for the two physicians betveen 56-60 years of age who had 
more patient contacts per Gay than would be expected. 

When a similar tabulation is done for only the General Practitioners in the 
study, a similar curve result:*, including the variation in age group 56-60 since 
both of the physicians involvcJ are General Practitioners. 
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The statistics dealing with patient volumes by type of practice are not so 
definitive, when all 76 participants are concerned: 



Table 18 

Average Daily Patient Contacts by Type of Practice 



Solo 



Unispec. 
-5 

Members 



Mixed 
-5 

Members 



Unispec, 
5+ 

Members 



Mixed 
5+ 

Members 



No, of Fhys. 8 22 4 5 

Ave. Daily Con. 67 45 39 61 

However, when similar tabulations are done for only the 63 General 
Practitioners, a pattern does result: 



37 
45 



Table 19 

Average Daily Patient Contacts by Type of Practice 
~~ ~ (General Practice) ~ 



Solo 

No. of Phys. 8 
Ave. Daily Con. 66 



Unispec. 

. -5 
Members 

22 

45 



Mixed 
-5 

Members 
4 
39 



Unispec. 

Members 
5 

52 



Mixed 
5+ 

Members 
24 
50 



It would appear that those General Practitioners in solo or large group 
practice have a greater volume of patient contacts in a day than those in small 
group practice, and those in group practice with other General Practitioners have 
slightly higher volumes than those 4n group practice with specialists. The small 
physician sample in some categoric^ may affect the validity of these conclusions; 
it must be confirmed by further data collection with a larger sample. 

There are variations when one compares the method or place of patient 
contact with the medical specialty of the physician, but none are particularly 
surprising. For example, the one Surgeon profiled has 63% of his practice in 
the hospital, as compared with an average of 19.3% for all physicians, and a 
proportional decrease in office contacts. The Internists also show a higher than 
average use of the hospital, 29.4%, while Pediatricians use the telephone for 
26.0% of patient contacts compared with an average of 18.4%. 
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Concerning age of patients, there are also variations, but: none unexpected. 
Pediatricians have the predominant portion of their practices in the 0-14 age group, 
while Internists and the Surgeon primarily serve a patient population over 40 years 
of age. General Practitioners serve all age groups. 
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Testing 

Tests were administered to 63 of the 76 participants, as follows; 



Table 20 

Test Administration 
(by number of participants tested) 

Number 

General Practice 52 
Pediatrics 7 
Internal Medicine 3 
Surgery 1 
Total 63 

Tests were sent to the other 13 participants, but had not been returned by 
the end of the study period. 

As previously indicated, the testing procedure proved to be the weakest link 
in phase 1 of the study. The problems encountered were: (1) limitations in the 
scope" of the test bank, (2) limitations in the structure of the bank, (3) inappropri- 
ateness of many test questions to clinical practice, and (4) high cost of testing. 
In general, these have been resolved in phase 2. 

While the number of questions in the new test bank for the second phase is 
not significantly higher than in phase 1, the quality and distribution evidently was 
adequate. The clearest indication of this is that the test bank was generally 
successful in fulfilling the requirements of the test formula; lack of questions 
either in volume or distribution within the ICDA would have resulted in observable 
differences between the practice profiles and tests. 

Expanding the classification system from the 18 major ICDA categories to 
the most definitive 975 classifications apparently solved the structural problems. 
In this way, it is possible: to virtually assure a physician that each question in 
his test will be directly related to one or more diagnoses he has reported. There 
are some remaining problems related to testing medical specialists; e.g., with a 
classification system oriented to diseases and conditions it is not possible to test 
effectively in such areas as anesthesiology, radiology, etc. 
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The cycling of questions through clinicians apparently solved the problem of 
inappropriateness. This was determined by asking some participants to review 
each question asked them and judge whether it was relevant to their practice. In 
phase 1 this resulted in a 66.7% relevancy rate; in phase 2 this rose to 88.3%. 

Cost of the procedure was significantly reduced by eliminating the remote 
teletype terminal used in phase 1. However, hand preparation of the tests and 
tabulation of the results involved considerable staff time. Since this is an area 
that lends itself to automation, hopefully the costs of this procedure could be 
further reduced by computerization. 

Test Formula 

While the test formula, previously described, accomplished the major ob- 
jectives set for it, it was not totally successful. Analysis of practice profiles 
and the corresponding tests indicates a high level of success in matching test 
composition to the volume and distribution of patient contacts and diagnoses. How- 
ever, there appears to be an incompatability between the goals of testing broadly 
on the practice profile and at the same time providing the physician and consultant 
with test results in sufficient depth to permit judgments tc be made. 

The formula is written to provide from one to five questions in any one of 
the 975 sub-sub-categories; seldom would more than five questions be asked. 
Experience of the consultants is that the present maximum of five should probably 
be the minimum; they find it difficult to make judgments on less. However, if 
the length of the test is to remain constant at 125 questions, taking approximately 
two hours, this would limit testing to a maximum of 25 of the 975 sub-sub-cate- 
gories in the ICDA. 

The following table presents the impact of the attempt to obtain broad 
coverage of the profile on the testing results. The first column indicates the 
percentage of diagnoses covered by the entire test, and the second column the 
percentage of diagnoses covered when those major categories in which less than 
10 questions were asked are eliminated: 
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Table 21 

Coverage of Practice Profile by Test 



(by per cent) « 

Percentage 

Percentage Covered by 

Covered by In-depth 

Total Test Testing 

General Practice 90.4 66.0 

Pediatrics 83.0 68.0 

Internal Medicine 96.3 67.8 

Surgery 92.0 80.0 

Total 91.3 66.6 



Other Problems in Test Administration 

Another problem area involves idiosyncracies in the classification system. 
Category 16, "Symptoms and Ill-defined Conditions, " accounted for a substantial 
number of diagnoses and consequently had a major influence on test composition. 
While it was possible to test in this category, the results were of limited use to 
the participants and consultants. It has only 17 sub-sub-categories, and some, 
such as 16-B-791 ''Headache," are useful. However, most merely refer back to 
one of the other 17 major categories, e.g., 16-A-783,' 1 Symptoms Referable to 
respiratory system.' 1 The solution may be to change coding procedures so that 
all diagnoses in category 16 are related to the other categories and used as 
contributing data to any educational determinations in that way. 

A second idiosyncracy in the classification system involves category 17, 
"Accidents, Poisonings, and Violence. " This single category has 187 of the 975 
sub-sub -category listings in the ICDA; it is much more definitive than any of the 
other 17 major categories. The result is that the total number of diagnoses in 
category 17 may be high, but the number in any one sub-sub-category is likely 
to be quite low. The test formula was written to attempt to provide for this 
problem, but examination of the data indicates that it was only partially success- 
ful; less testing was done in category 17 than was indicated by total volume of 
diagnoses. Even when testing was done, the classification was too specific to 
lend itself to educational consultation, e.g., one classification, 17-1-882, deals 
with M Open wound of hand except finger(s)," and another, 17-1-883 deals with 
"Open wound of finger(s)." A combining of sub-sub-categories may solve the 
problem for the future. 
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A further possible improvement in the testing procedure involves M weighting' 1 
of diseases and conditions so that those that are more life -threatening, or likely 
to result in disability, play a greater role in test composition than others. Ex- 
periments are now being carried out to determine the feasibility of this. 

Finally, the new testing procedure of using written tests has resulted in an 
administrative problem. In phase 1, with an interactive test administered on-line 
with the computer, the participant had to set aside a specific time period in 
which to complete his test. The written test takes its place among other priorities 
in his busy schedule. This has resulted in the late return of tests (an incon- 
venience and delay in the process) and non -return of tests in 13 instances, which 
makes continuation of the educational planning procedure impossible. If this is a 
problem with selected, highly-motivated participants, it may become a serious 
handicap when dealing with a general physician population. 

Test Scores 

There is little value or validity in considering cumulative test scores; each 
test was of different composition so comparison on a meaningful basis is difficult. 
Consequently, the discussion of cumulative scores will be limited. 

One requirement was that the individual questions, and the consequent test, 
be sufficiently "difficult" to provide the necessary discrimination on which to base 
decisions. A test on which most of the questions are answered right, or most 
wrong, would not meet the requirements of the study. Ranges of the test scores 
for the physicians tested were: 

Table 22 

Ranges of Test Scores 
(by number of participants) 

Per cent Correct 



Less 


50% 


60% 


70% 


More 


Than' 


to 


to 


to 


Than 


50% 


59% 


69% 


79% 


80% 


2 


5 ., 


23 


23 


10 



Number of 
Participants 

In th^ judgment of the project staff most of the tests did fall into a range 
where there was sufficient discrimination on which to base recommendations and 
decisions. Where tests did not render useful data it usually involved a physician 
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who scored in excess of 80% correct; the few incorrect answers did not present a 
basis for judgment. 

Test Analysis 

It is important to note that for the purposes of this study, the physician is 
in competition with himself on the test, not with other physicians. Therefore, it 
is not important that the total test scores do not lend themselves to accepted 
forms of statistical analysis. Each physician's total test score is used as a base, 
with which scores in specific segments of the test are compared, e.g., if his 
score in "Respiratory Disease" is a significant number of percentage points below 
his total score, it is an area which warrants further investigation. 

The procedure for test analysis which evolved during the second study phase 
is as follows: 

1. Test results in each of the 18 categories are compared with the total 
score and with each other to determine if general areas can be identi- 
fied for more detailed analysis; this is essentially a screening procedure. 

2. Once the major category or categories have been identified, examination 
is made of each of the sub-sub-categories included in that major cate- 
gory. This provides information on whether those questions answered 
incorrectly are generally distributed throughout the category, or related 
to specific diseases or conditions. 

3. Finally, if the above steps appear to identify probable or possible areas 
for study, the individual questions asked on the test give further infor- 
mation, particularly those answered incorrectly. 

This would lead to the analysis of one physician's test results as follows 
(total score was 71.2%): 
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Table 23 
Report of Test Results -#24093 





No. of 
uiagnoses 


% of 
Practice 


No. of 
v^iic scions 


Number 
Right 


Number 
Wrong 


Per cent 
Kignt 


lo 


7U 


or o 07 
OD. 2% 


1*7 


1 A 
10 


1 1 

1 1 


en Q 07 

59. 6% 


1 / 




1U. 1% 


lo 


1 1 


c 
D 


AQ O 07 
OO. 0 7b 


Q 
O 


1 O 


Q Q 07 

o. OTb 


1 / 


Q 

y 


Q 
O 


CQ O07 


7 


10 


*7 r» (J7 

7.9% 


15 


lo 


2 


86. o% 


10 


lo 


A n07 


1 A 

lo 


A4 


2 


87. 5%) 


1 

1 


lz 


5. 5% 


3 . 


2 


1 


66.7% 


10 


1U 


4.6% 


7 


0 


i 


85.7% 


A 
0 


y 


4.0% 


4 


4 


0 


1 /"\/"\ /> fT7 

100. 0% 


12 


9 


4.0% 


4 


2 


2 


50.0% 


3 


8 


3.7% 


8 


6 


2 


75.0% 


2 


6 


2.7% 


0 








5 


5 


2.3% 


3 


3 


0 


100. 0% 


9 


5 


2.3% 


0 








11 


5 


2.3% 


4 


2 


2 


50. 0% 


13 


3 


1.3% 


0 








14 


2 


0.9% 


0 








4 


1 


0. 5% 


0 








15 


0 


0. 0% 


0 









The first observation that can be made is that there was testing in reason- 
able depth in five of the 18 major categories. It is of interest to note that the 
computer, following the test composition formula, had proceeded through all 18 
categories and assigned test questions in any sub-sub-category where there were 
three or more diagnoses; it had completed the same procedure and assigned 
questions in all sub-sub-categories where there were two diagnoses. Since the 
number of questions still did not reach the limit of 125, it began to repeat the 
process for those sub-sub-categories with only one diagnoses and had proceeded 
through categories 18, 17, 8, 7 and 16 before it reached the limit. 

The second observation is that in view of the participant's total score of 
71.2%, tnere are two categories where further analysis is warranted: category 
18 with a score of 59.3% and category 8 with a score of 53.9%. The other three 
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categories in which a significant volume of testing took place have scores at or 
significantly above the total score. 

Detailed analysis of category 18 reveals the following: 

Table 24 

Category 18 Test Results - #24093 



Sub -sub - 




Total 


No. 


No. 


% 


Category 


Disease or Condition * 


Questions 


Right 


Wrong 


Right 


18-A-005 


Well baby examination 


5 


3 


2 


60% 


G-060 


Pre-natal care 


5 


3 


2 


60% 


H-070 


Post-partum care 


5 


3 


2 


60% 


C-020 


Inoculation & vaccination 


5 


3 


2 


60% 


A -000 


General examination 


4 


2 


2 


50% 


J -090 


Contraception 


3 


2 


i 


60% 



It would appear the physician is equally prepared over the entire category, 
but scores less well than in other areas of medicine. Consequently, he and the 
consultant will examine the 27 questions asked, with particular attention tto the 11 
he answered incorrectly, as one step in making the educational diagnosis. 

Another physician with a similar profile and a slightly lower total test score, 
had the following results in the same category: 

Table 25 





Category 18 Test 


Results - 


#26044 






Sub -sub - 
Category 


Disease or Condition 


Total 
Questions 


No. 
Right 


No. 
Wrong 


/o 
Right 


18-A-000 


General examination 


5 


4 


1 


80^ 


G-060 


Pre-natal care 


5 


1 


4 


20 % 


A-005 


Well baby examination 


5 


3 


2 




A -009 


Unspecified examinations 


1 


1 


0 


ioo c : 


A-002 


Radiologic examination 


2 


2 


0 


100% 



In this instance it is doubtful the physician requires study in the entire cate 
gory; the discussion between the consultant and participant would center on " Pre- 
natal care," and perhaps "Well-baby examination, " While this may appear a bit 
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detailed, and based on a small amount of data, it should be pointed out that the 
physician reported 10 patient contacts for M Pre-natal care" during the four days 
data were collected, an average of 2.5 per day. If this is an accurate reflection 
of his practice, and he sees patients on 240 days in a year, it indicates 600 
patient contacts in this sub -sub -category annually. 

Analysis of test data also included examination of results in those major 
categories where few questions were asked. An example of this is presented by 
category 3 where there were seldom enough diagnoses to warrant a high volume 
of questions. However, when testing did occur, it was usually in the sub-sub- 
categories involving "Diabetes" and "Obesity," There were frequent occasions 
when four or five questions were asked in either of these two sub-sub-categories 
and few answered correctly. These at least represented areas warranting dis- 
cussion even though few questions were asked. In the final analysis, the only 
difference between a result of answering two questions correctly out of five and 
eight correctly out of 20 is in the degree of confidence one has in the result; 
both indicate 40% success. 

It should be pointed oul that the above procedure evolved during phase 2, 
and was not used throughout the study. In the majority of the test analyses, test 
results were given in the 54 categories in which the Educational Resource Index 
is classified, on the rationale that this would permit direct transfer of test data 
to that index for selection of educational events or materials. The result was 
that it was extremely difficult for the consultant and physician to get an over- 
view by major category, since the data were presented in a fragmented manner. 
This will be discussed in detail later, since it had an effect on usefulness of test 
data in the consultation process. 

Diagnostic Use of Test Results 

The procedure described above v/as applied to test results of the 63 
physicians tested, for the purposes of this report, even though it was not used 
on all of them in carrying out the stuo M This was done because it gives a more 
realistic measurement of the role of testing in the total process than was actually 
achieved. 

This procedure would give the consultant one of the following results: 
(1) identify probable areas for study, (2) identify possible areas for study, (3) 
identify no areas. 
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In the first instance, this might involve entire categories where a significant 
volume of questions were asked and the scores in those categories were substan- 
tially below his total score, or it might involve specific sub-sub-categories v/here 
few questions were asked but all, or almost all, were answered incorrectly. 

Probable areas of study involved those in categories where the total score 
was only slightly below the total test score, or individual sub -sub-categories re- 
vealed two or three questions out of five answered incorrectly. 

Criteria for making these judgments varied from one test to the next since 
each judgment had to be made relative to the individual's performance on other 
test segments rather than against a standard. 

The third possibility was that the test results revealed no area of potential 
study. This occurred most often with those who achieved high total scores, and 
consequently had few incorrect answers on which to base determinations. It also 
occurred with some physicians who had medium or lower level scores, but scored 
equally well across the entire profile. 

The results of the 63 participants tested are as follov;s: 

Table 26 

Indications for Study Based on Testing 
(by numbers of participants) 

Indicated Indicated Indicated 

Probable Possible No Area 

Study Area(s) Study Area(s) of Study 

General Practice 20 26 6 

Specialty Practice 1_ 7 3 

Total 21 33 9 

In those instances where test results indicated both probable and possible 
areas of study, the physician is listed under "probable" since this is considered 
the most valuable result. 

As indicated in the above table, 88.5% of the general practitioners and 
72.7% of the specialists received some indication of educational need from the 
test results. There is a difference in the degree of indication. For general 
practitioners, 38,5% received probable indications and 50.0% possible indications. 
However, for specialists only 9.1% received probable indication of educational 
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need and 63.6% received possible indications. The testing procedure was signifi 
cantly less definitive for them. 

Volumes of Potential Study Areas Identified 

Another measurement of the value of the test data is in the volu:*ne of in- 
formation provided to each participant tested. 

Table 27 

Probable Areas of Study Identified 





Entire 




Sub-sub- 










Categories 


Ave. 


Categories 


Ave. 


Total 


Ave. 


General Practice 


18 


0.3 


30 


0.6 


48 


0.9 


Specialty Practice 


1 


0. 1 


0 


0.0 


1 


0. 1 


Total 


19 


0.3 


30 


0.5 


49 


0.8 



As indicated, 49 areas of probable study including total categories or 
specific sub-sub-categories, were identified for the 63 physicians, an average of 
0.8 areas per physician. The range is from zero to six. Considering only 
general practitioners, the average is 0.9, with the same range. For specialists 
the identification of probable areas of study was almost totally unsuccessful. 

Table 28 





Possible 


Areas 


of Study Identified 










Entire 
Categories 


Ave. 


Sub -sub - 
Categories 


Ave. 


Total 


Ave. 


General Practice 


30 


0.6 


134 


2.6 


164 


3. 1 


Specialty Practice 


4 


0.4 


34 


3. i 


38 


3.5 


Total 


34 


0.5 


168 


2.7 


202 


3.2 



With 202 possible areas of study identified for the 63 participants, the 
average is 3.2, with a range of zero to nine, 

Considering the data in both tables 27 and 28, it would appear that the test- 
ing mechanism identified both probable and possible areas of study for general 
practitioners in reasonable volumes and at both the category and sub-sub-category 
level, with emphasis on the latter. For specialists, the volume of probable areas 
of study was virtually zero, and the most guidance was in possible areas of study 
at the sub-sub-category level. 



49 



Specific Areas of Study Identified 

Accepting the limited sample of physicians involved, and the variance in test 
composition and analysis, some general observations can be made concerning the 
specific areas of study indicated by the testing procedure. 

Combining the data in tables 27 and 28, one gets the following results: 

Table 29 



Probable 


and Possible Areas 


of Study Identified 






Entire 


Sub-sub- 






Categories 


Categories 


Total 


General Practice 


48 


164 


212 


Specialty Practice 


_5 


34 


39 


Total 


53 


198 


251 



Distribution of these 251 areas within the 18 major categories of the ICDA 
is as follows: 

Table 30 







Distribution of 


Areas of Study Identified 








p General Practice 


j- Specialty Practice-] 


1 


Total 


ICDA 


Entire 


Sub -sub 


Entire Sub- sub - 


Entire 


Sub-sub- 


Category 


Cat. 


Cat. 


Cat. Cat. 


Cat. 


Cat. 


1 






2 




2 


2 


4 


2 




4 


2 


3 


2 


22 


1 . 1 


3 


23 


4 






1 




1 


5 


1 


7 


2 


1 


9 


6 


1 


1 


1 1 


2 


2 


, 7 


12 


35 


3 


12 


38 


8 


9 


24 


2 


9 


26 


9 


2 


7 


1 1 


3 


8 


10 




8 


1 




- 9 


11 












12 


1 


4 


J 2 


2 


6 


13 


1 


4 




1 


4 


14 






1 




1 


15 






2 




2 


16 


1 


2 




1 


2 


17 


3 


9 


2 


3 


11 


18 


n 


39 


1 I! 


12 


52 




48 


164 


5 34 


53 


198 
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A comparison of the areas of potential study- identified by testing for the 52 
general practitioners with the volumes of diagnoses reported in the 18 ICDA 
categories renders interesting results: 

Table 31 

Comparison of Volumes of Diagnoses 
and Potential Study Areas Identified by Testing 
" (52 General Practitioners) 

Rank Order by ICDA Category 



Category Category 

By Volumes By Volumes of 

Rank of Potential 

Order * Diagnoses Study Areas 



Most 18 18 

Frequent 8 7 

7 8 

16 3 

17 17 

10 9 

3 5 
9 10 

5 2 

6 12 

12 13 

13 16 
2 6 
1 1 

4 4 

11 11 
Least 14 14 
Frequent 15 15 



A similar comparison for specialists is unproductive, since only category 
18 has a significant volume and the other 17 categories were identified as potential 
study areas from zero to three times each. 

Table 31 indicates that the greatest variations occur with category 2 which 
rises four levels in the rank order, category 3 which rises three levels, and 
•categories 5 and 9 which rise two levels, as well as category 16 which drops 
eight levels, category 6 which drops three and category 10 which drops 2. In 
general, the correlation between volume of diagnoses and volume of identified 
study areas is quite close* 
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A more detailed analysis, by category and sub-sub-category of the 52 
general practitioners of the areas of potential study as indicated by testing is: 

Table 32 

Areas of Study Identified by Testing 



ICDA 



Category 


Subject Area 


Number 


18 


Special Conditions and Examinations 


11 


A -000 


General medical examination 


7 


A-002 


Radiological examination 


2 


A -003 


Laboratory examination. 


1 


A-004 


Well baby and child care 


7 


C-020 


Inoculation and vaccination 


2 


G-060 


Prenatal care 


9 


H-070 


Postpartum observation 


2 


J- 090 


Contraception and sterilization 


2 


K-100 


Medical aftercare 


2 


K-105 


Surgical aftercare 


5 



Total 50 



7 


Diseases of the Circulatory System 


12 


A-390 


Rheumatic fever 


2 


C-401 


Benign hypertension 


13 


D-4 10-14 


Ischemic heart disease 


6 


E -420 -29 


Other forms of heart disease 


7 


F -430 -38 


Cerebral vascular disease 


2 


G -440 -58 


Circulatory system 


5 


Total 




48 


8 " '• 


Diseases of the Respiratory System 


9 


A -462 


Acute pharyngitis 


2 


A-465 


Acute upper respiratory infection 


3 


A-466 


Acute bronchitis 


2 


C-480 


Viral pneumonia 


1 


C-486 


Pneumonia, unspecified 


5 


D-490 


Bronchitis, unspecified 


6 


D-491 


Chronic bronchitis 


1 


D-493 


Asthma 


2 


E-507 


Hay fever 


2 


Total 




33" 


3 


Endocrine, i Nutritional and Metabolic 


2 


A-240 


Endocrine 


1 


B-250 


Diabetes mellitus 


12 


D-277 


Obesity 


9 


Total 




2T 


17 


Accidents, Poisoning and Violence 


3 


A-800-29 


Fractures 


1 


E -840-48 


Sprains and strains 


3 


Q-960-79 


Adverse drug effects 


4 


S-990-99 


Other adverse effects 


1 






12 



9 
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ICDA 

C ategory Subject Area Number 

9 Diseases of the Digestive System 1 
B-530-37 Esophagus, stomach and duedenum ' 3 
F -570*77 Liver, gallbladder and pancreas 5 

Total T 

5 Mental Disorders 1 
B-300 Neuroses 6 
B-303 Alcoholism 1 

Total , 8 

10 Diseases of the Genitourinary System 

A-580 . Acute nephritis 1 

B-599 Diseases of urinary tract 2 

E-625 Diseases of the uterus 2 

E-626 Disorders of menstruation 3 

Total 8 

2 Neoplasms 4 

A-F Malignant neoplasms 1 

H-221 Benign neoplasms of female genital organs 1 

Total "TT 

12 Diseases of the Skin and Subcutaneous Tissue 1 
B-692 Eczema and dermatitis 3 
O708 Urticaria J_ 

Total 5 

13 Diseases of Musculetal System & Conn. Tissue 1 
A -7 12 Rheumatoid arthritis 2 
A-715 Arthritis, unspecified 1 
B-720 Osteomyelitis 1 

Total 5 

16 Symptoms and Ill-defined Conditions 1 

A-784 Symptoms referable to upper gastroint. Tact 1 

B-791 Headache 1 

Total ~3" 

6 Diseases of the Nervous System Si Sense Organs 1 
E-G Diseases of the eye and ear 1 

Total 2 



There were no testing indications for study in categories 1, 4, 11, 14 and 

15. 

If testing, in itself, is accepted as a method of identifying educational needs, 
the above * iting would be of value in planning continuing education programs and 
product: a of continuing education materials. 
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Indications for programming would be: 

Special Conditions and examinations without illness, with emphasis on general 
medical examinations, well baby and child care, and prenatal care. 

Diseases of the circulatory system, with emphasis on hypertension. 

Diseases of the respiratory system, with emphasis on bronchitis and 
pneumonia. 

Diabetes ani obesity. 

Neuroses. 

For the purposes of this study, however, the test results were utilized as 
only one factor, albeit a major one, in the consultation process. 
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Educational Consultation 

During phase 1 of the study a variety of faculty members served as educa- 
tional consultants to the 37 participants. While they demonstrated a high degree 
of interest and involvement, there were two major handicaps which decreased their 
effectiveness: (1) unfamiliarity with private clinical practice, and (2) unfamiltarity 
with educational resources available. For these reasons, it was determined that 
in phase 2, consultants would be limited ro a small number who had a background 
of private practice and were completely familiar with the processes and adminis- 
trative procedures of the study. 

Three educational consultants, all faculty members, were involved in phase 
2. The majority of the consultation was done by a specialist in Internal Medicine 
with 14 years' experience in private practice before joining the faculty. He was 
assisted by a General Practitioner and a Pediatrician with 10 and two years 
experience in private practice, respectively, 

Consultation Procedure 

The consultation procedure was a flexible one, within a general format, 
which varied according to the value of the objective data available, the personality 
and experience of the consultant, and the personality and background of the partici- 
pant. 

When the participant had completed the testing, the consultant would be fur- 
nished with the objective data concerning the physician, his practice setting, his 
practice profile, and the test results. After review of these,, he would visit the 
participant, usually in his office setting. While the actual consultation was flexi- 
ble, it followed a general pattern: 

1. The consultant would review the data regarding the physician and his 
practice setting with the participant and determine if he could leave his 
practice for continuing education, and if so how long and how often. 

2. They would then jointly review the patient data and cross tabulations of 
it. 

3. Next came a discussion of the practice profile, based on a bar-graph 
presentation of diagnoses in the 18 categories of the ICDA. 
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4. Then they would jointly examine the test results, with discussion 
centered at the sub- sub-category level and in the areas with the greatest 
volumes of patient contacts, based primarily on those questions answered 
incorrectly. This included review of specific questions. 

5. The consultant would then ask the questions listed on the consultant's 
report (Exhibit 7). 

6. He would then demonstrate the Educational Resource Index so that the 
participant would be able to effectively utilize those portions eventually 
sent to him. 

During this entire process, which generally took about, two hours, the con- 
sultant would take detailed notes on the participant's comments and observations. 
It was found that if these were taken as direct quotations they were most useful 
for future determinations. 

Upon return to the central office, the consultant would then review all of the 
information and data and direct the staff in preparation of a final report to the 
physician. This would include: 

1. A summary of physician data and general patient data. 

2. The practice profile. 

3. Suggestions on implementation of a recommended continuing education 
program. 

4. Appropriate sections of the Educational Resource Index. 

5. A personalized letter summarizing the results of the consultation. 

Consultations Completed 

Of "he 76 participants, 63 completed the testing phase and 58 of these were 
consulted as follows: 
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Table 33 



Educational Consultation Completed 
(by number of participants) 



Specialty 
General Practice 



Number 



Pediatrics 



5 



Internal Medicine 



1 



Surgery 



1 



Total 



58 



Analysis of Consultation 

As one examines the process of educational consultation as it evolved in the 
second study phase, it appears there were four major factors that contributed to 
a greater or lesser degree to the design of individual educational programs. 

Two of these were based on objective data: (1) Practice profile (number of 
diagnoses by ICDA categories), and (2) Test results. 

Two were based on subjective information: (1) Interpretation of test results 
by the interaction of the consultant and participant. (This differs from the ob- 
jective analysis above in that the perceptions of the participant and subsequent in- 
sights by the consultant added significantly to the simple statistical analysis.), 
and (2) Interaction between the consultant and participant concerning the physician's 
practice situation, perception of educational needs, and personal characteristics. 

By examining each of the four factors in relation to the final educational 
program suggested, it is possiblej^ make judgments on which factors apparently 
contributed to the end result. 

In the analysis, the various elements are: 

Factor A: Practice profile . If the category or sub-sub-category designated 
as an area for study correlates with a significant number of diagnoses in the 
practice profile, it is assumed for purposes of the tabulation that the profile was 
a significant factor in the selection. In certain instances the educational pre- 
scription is too general to relate to the profile; in others it is in an area where 
few or no diagnoses were recorded. 
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Factor B: Test Results. Statistical analysis, with some judgmental de- 
cisions as described previously in t\As report, was used retrospectively to identify 
categories and sub-sub-categories in which the testing indicated probable or possi- 
ble areas of study. These were correlated with the designated areas of study, 
and when there was correlation it was assumed that the test data did play a role 
in educational diagnosis. In some instances education was recommended in areas 
where no testing occurred, and in rare instances there were educational recom- 
mendations contrary to the test results. 

Factor C: Interaction on Test Results. When the test results were used to 
motivate discussion between the participant a? J consultant, the results were suffi- 
ciently substantial and subjective to constitute a major input to the final educational 
diagnosis. For this reason, this interaction is considered separately from Factor 
2, above. These results were recorded by the consultant and when they corre- 
lated with the educational diagnosis it was assumed they played a major role in 
design of the educational program. 

Factor D: Interaction on Practice Characteristics . This represents dis- 
cussion by the consultant and participant on a variety of subjects, e.g., the 
physician's perceived needs in continuing education, problems he encounters in 
his practice, his likes and dislikes, for example. Decisions based on these fac- 
tors were tabulated in the consultant's report, and when these correlated with the 
educational program recommended, they were assumed to have played a major 
role in its composition. 

Process Analysis 

A total of 170 specific educational recommendations were made to the 53 
physicians who completed the consultation procedure. This ranged from one to 
six recommendations per physician, with an average of 2.93. Each educational 
recommendation made may have been the result of from one to all four of the 
factors previously listed; the distribution is as follows: 

Table 34 

Number of Factors. Contributing to Educational Diagnoses 





Volume 


of 


Factors 


Involved 




J_ 


_2 


_3_ 


_4 


General Practice 


49 


48 


50 


10 


Specialty Practice 


_6 


_3 


J_ 


_3 


Total 


55 


51 


51 


13 
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Under ideal circumstances axi educational diagnosis would be the result of 
all four factors previously defined; this occurred only 13 times out of 170 (7.6%). 
Decisions based on three, two, or one factors occurred with, about equal frequency 
(approximately 30% each). The general contribution of each of the four factors 
can be tabulated as follows: 

Table 35 



Contribution 


of Factors to 


Educational 


Diagnoses 






(by per 


cent) 










Specific Factors* 






_A_ 


B 


_C_ 


D_ 


General Practice 


68.6 


45.8 


48.4 


51.6 


Specialty Practice 


52.9 


35.3 


52.9 


52.9 


Total 


67. 1 


44.7 


48.8 


51.7 



* Factor A - Practice Profile 
Factor B - Test Results 

Factor C - Interaction Based on Test Results 

Factor D - Interaction Based on Practice Characteristics 



Based on this correlation, it would appear that the practice profile (volumes 
of diagnoses made in ICDA categories) had the greatest influence on educational 
diagnosis for general practitioners with the contribution of the other three factors 
being approximately equal. For specialists the statistical analysis of test results 
was least useful, with the other three factors being equal. 

Inter-relationships of Factors 

Having identified the relative roles of the four factors in arriving at 
recommended educational programs, it is worthwhile to determine in greater de- 
tail the specific roles and inter-relationships of them in the consultation process. 

When the educational diagnosis was based on single or multiple factors, the 
distribution among the four factors identified was: 
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Table 36 

Recommendations Based on One factor 



1 




Specific 


Factors* 






A 


B 


C 


D 


Total 


General Practice 


3 


1 


3 


42 


49 


Specialty Practice 


1 


J_ 


J_ 


_3 


_6 


Total 


4 


2 


4 


45 


55 



Table 37 

Recommendations Based on Two Factors 

Specific Factors* 





A & B 


A & C 


A & D 


B & C 


B & D 


C & D 


Total 


General Practice 


8 


16 


19 


2 


1 


2 


48 


Specialty Practice 


g 


_3 


_0 


0 


0 


0 


__3 


Total 


8 


19 


19 


2 


1 


2 


51 



Table 38 

Recommendations Based on Three Factors 



General Practice 
Specialty Practice 
Total 



A-B-C 
42 
J_ 
43 



Specific Factors* 
A-B-D A-C-D B-C-D 



7 
0 
7 



1 

i 



o 

0 
0 



Total 
50 
_1 
51 



Relative Value of Educational Diagnoses 

The prior analysis of the number of factors involved in a single educational 
diagnosis, and their inter -relationships, permits judgments to be made as to the 
value of each diagnosis. 



* Factor A - Practice Profile 
Factor B - Test Results 

Factor C - Interaction Based on Test Results 

Factor D - Interaction Based on Practice Characteristics 



9 

ERIC 



60 



Ideal Result - The 13 situations in which all four factors evidently had a 
role in the educational diagnosis are considered the best possible result in terms 
of the research objectives* 

Effective Result - The 51 instances where three of the four factors were in- 
volved also appear to effectively meet the research objectives. In the majority 
of these instances the two objective factors were combined with the subjective con- 
clusions of the consultant and participant, based on test results. In all but one of 
the remainder the objective data were combined with interaction concerning the 
physician's practice characteristics. Only in one instance was the diagnosis based 
primarily on subjective data. 

Moderately Effective Result - It is somewhat difficult to place a value judg- 
ment on the 51 educational prescriptions based on two of the four factors. As 
indicated in Table 37, the majority of these included the practice profile as one 
of the factors, so at least any prescribed area of study did relate to diseases and 
conditions which the physician was called on frequently to treat. The other factor 
involved was usually subjective, based on interaction between the consultant and 
physician. 

Ineffective Result - Of particular concern are the 55 instances where the 
educational prescription was based on only one of the four factors. As indicated 
in Table 36, this factor was usually interaction by the consultant and participant 
over his practice characteristics. 

Further analysis indicates that some of these indicate possible areas of 
weakness in the process; others are relatively unimportant. 

The first is a group of 25 diagnoses made for physicians who already had a 
planneu educational program in one or more areas of study based on results 
classified above as ideal, effective, or moderately effective. The diagnoses 
based on only one factor were in addition to the acceptable educational plan, and 
consequently are not of great concern. 

Another group of 22 can be traced to tests which did not give clear indica- 
tions of potential areas of study. Consequently, the only factors available to the 
consultant and physician were the practice profile and interaction based on prac- 
tice characteristics. Usually one or more educational diagnoses were made on 
the two factors, and those based on a single factor were supplementary. 




Finally, there were eight educational diagnoses where the data proved of 
little value in identifying educational needs, and the consultant and participant had 
no option but to reiy on only one factor. 

In summary, by placing value judgments on the 170 educational diagnoses 
made, one arrives at the following: 

Table 39 
Value of Educational Diagnoses 



Value Number 
Ideal Result 13 
Effective Educational Diagnosis 51 
Moderately Effective Diagnosis 51 
Ineffective Educational Diagnosis 

Supplementary to Effective Diagnoses 25 
Due to High Test Scores 22 
Due to Unknown Reasons 8 
Total J35 
Total * 170 



With nearly one-third of the diagnoses based on only a single factor, and, 
consequently considered ineffective in terms of program planning, similar value 
judgments can be made by examining their distribution among the 58 physicians 
involved. 

For this purpose, it is considered effective program planning if the partici- 
pant has one or more educational diagnoses based on three factors, moderately 
effective if this is reduced to two factors, and ineffective if there was no educa- 
tional diagnosis based on more than one factor. A tabulation on this basis gives 
the following results: 

Table 40 
Value of Educational Programs 



Value Number 
Effective Educational Program Planning 35 
Moderately Effective Program Planning 18 
Ineffective Program Planning 5 
Total 58 
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In summary, as a result of the process, it was possible to design acceptable 
educational programs for 53 of the 58 participants. The remaining five may be 
indicative of a failure of the process, or may identify physicians who have effective 
continuing education programs and consequently do not have detectabl< areas of 
weakness. 

Problems and Preferences in Practice 

As indicated, there were a number of instances in educational diagnosis when 
the data gave little or no guidance, and decisions were based on practice charac- 
teristics, including tiie likes and dislikes of participants. 

It is of interest to note that of the 58 participants consulted, 26 listed one 
form or another of mental disorders as the problem which caused them most dis- 
tress. Seven listed obstetrical complications as the major problem area and four 
listed geriatrics. 

On the positive side, 17 listed obstetrics as the most enjoyable part of 
medical practice; 12 listed surgery and 10 listed pediatrics. 

Time as a Barrier to Continuing Education 

Recognizing time, in relation to patient demands and responsibilities, as a 
possible barrier to implementation of a continuing education program, the con- 
sultants attempted to determine how free physicians were to leave their practices 
for short periods of study. 

Forty-eight physicians responded to this inquiry, and gave the responses on 
the following page. (Table 41) 

This ability to leave the demands of practice for a week or more by 40 of 
the 48 physicians who responded to the question undoubtedly reflects on the large 
number in group practice in the sample. It did give the consultant and partici- 
pant a good deal of flexibility in the design of educational programs since virtually 
any of the entries in the educational resource index could be accomplished in the 
times indicated. 
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Table 41 

Ability to Leave Practice for Education 



Number Number 

Number of of Times 

of Days Physicians Per Year 

21 1 2 

21 1 3 

14 13 1 

14 10 2 

14 3 3-4 
14 15 

10 1 1-2 

7 2 1 

7 5 - 2 

7 2 4-5 

7 1 6 

3 2 1 

3 3 2 

3 1 3 

3 1 4 

3 1 Unknown 



Preferred Method of Study 

It is an accepted fact in the field of education that individuals learn best in 
specific ways, or a combination of ways. With this in mind, the consultant posed 
the question to participants as to the way in which they thought they learned best. 
Of the 52 participants who responded to the question, 50 indicated that they did 
know how they learned best; the results are as follows: 

Table 42 



Preferred Method of Learning 
(by number of participants) 

Method of Learning 





Audio 


Video 


By 
Doing 


Audio/ 
Video 


Audio/ 
Doing 


Video/ 
Doing 


Other 


General Practice 


8 


9 


18 


1 


3 


1 


5 


Pediatrics 


0 


0 


2 


0 


1 


0 


1 


Internal Medicine 


1 


- °_ 


_0 


0 


0 


0 


0 


Total 


9 


9 


20 


1 


4 


1 


6 
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If a physician was unable to specify his preferred or most effective method 
of study , the consultant attempted to assist him in making such a determination. 
Recall or experience in the first two years of medical school, when much knowl- 
edge was transferred by audio or visual methods, as opposed to the final two 
year when emphasis was by observing and doing, often helped in such a decision. 

The information was of use in designing educational programs, since those 
who thought they learned best by audio methods could select conferences or tape/ 
recorded home study programs, while those who learned best visually might con- 
centrate on texts and journals. For those who learned by doing, the best 
approach might be to arrange apprenticeships. 

Educational Resource Index 

The ability to leave practice for short periods of study, and the method in 
which each participant thought he would learn best gave 'the consultant and physi- 
cians indications of the form in which their demands should be made on the educational 
resource index. The entire index was much to cumbersome and complex to use 
on an individual basis, and each physician was provided only with those portions 
which applied directly to his planned educational program. 

Once the above determinations were made, and requests made for sections 
of the index, it is of interest to note how successful the. resource wa3 in meeting 
these demands on it. 

Tabulation indicates that 123 sections were sent to the 58 participants in 
response to the 170 educational diagnoses made, or 72.4 per cent success in ful- 
filling requests. Of the 47 instances where no listing was sent, 15 involved staff 
errors and four involved instances where specific apprenticeships were arranged 
at the time of consultation and no selection was necessary. However, in 28 in- 
stances the index was unable to respond. In 15 of these, the prescription was 
too general (e.g., geriatrics, sports medicine) and not in the classification 
system. In 13 instances it involved techniques or procedures (e.g., venipuncture 
in infants, obtaining and interpreting arterial blood gasses) which were difficult 
if not impossible to retrieve from the index. Consequently the index was unable 
to respond in 16.5 per cent of the instances; in another 11,2 per cent the index 
could have fulfilled the request, but was not called on because of staff error or 
other arrangements making it unnecessary. 
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Practice Management 

While the entire procedure, and objectives of the study, relate to the 
scientific practice of medicine, data collected and insights of the consultants led 
to the conclusion that there were elements of practice organization and manage- 
ment involved which had major implications for the quantity and quality of health 
care delivered. Of the 58 participants consulted, the major recommendations 
made were: 

1. Management of incoming telephone calls 

Establish written protocol 29 
Improve present methods 4 

33 

2. Delegation of responsibility to ancillary personnel 

Establish protocol for specific tasks 25 

r 

Make 'changes in present protocol 7 

32 
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Evaluation 

Throughout this report an attempt has been made to pass judgment on the 
success or failure of each procedure involved, and explore other options of 
accomplishing the same task better or more efficiently. This continuing process 
analysis constitutes one form of evaluation, and a summary of the results is as 
follows: 

Data Collection 

1. Both methods of data collection rendered the desired data. 

2. Use of dictating machines proved more economical, and consequently 
is the method of choice. 

3. The procedure cannot be eliminated by ha zing participants predict their 
practice profiles. 

4. The procedure cannot be eliminated by devising standard profiles. 

Testing ; 

1. There is an unresolved conflict between the goals of testing broadly on 
the practice profile, and in sufficient depth for educational diagnosis. 
At present the procedure is more successful in breadth of testing, and 
less successful in depth of testing. 

2. The classification system needs refinement for testing purposes. 

3. Testing carried out was relevant to clinical practice. 

4. Testing generally proviued the required discrimination for General 
Practitioners, but was less successful for Specialists. 

5. Testing did assist generally in identification of educational needs, but 
usually failed to provide conclusive evidence. 

6. It is not possible to design meaningful standardized tests. 

7. It is possible to make only limited judgments on common educational 
needs of a physician population on the basis of cumulative test results. 

Consultation 

1. Staff physicians fully familiar with the process appear better able to 
provide educational consultation than faculty subject area specialists. 
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2. While all factors in the process appear to contribute to the educational 
diagnosis, the practice profile appeared to be the most useful. 

3. The consultation procedure, based on both objective and subjective infor- 
mation, appeared effective in designing educational programs. 

4. The educational resource index was generally successful in providing 
specific educational events or materials to meet identified educational 
needs. 

Judgments of Participants 

In order to determine participants' attitudes toward the procedure, the 58 
physicians who completed the consultation process were surveyed; 34 returned 
the questionnaires (Exhibit 9), They were asked to evaluate both the procedures 
involved, and the results achieved. 

Concerning process analysis, comparisons can be made between the judgments 
of the staff and those of the physicians. The dam are as follows: 

Table 43 

Value of Procedures to Participants 
(by per cent) ■ 

Practice Consult - 

Profile Testing ation 

Staff Analysis 67. 1 44. 7 58. 8 

Participant Survey 65.4 58. 8 61. 4 

Difference -1.7 +14.1 +2.6 

The figures for staff analysis were obtained from Table 35, an assessment 
of the contribution of the various factors to determination of the educational 
diagnosis. The figures for the participants' evaluation were obtained from the 
survey form, where they rated each procedure on a scale of 1 to 9, with a rating 
of 1 regarded as 100% and a rating of 9 as 0%. 

The value judgments placed on the practice profile and consultation procedure 
are quite comparable; the participants obviously placed a higher value on the 
testing procedure than did the staff analysis. 

Participants were also asked in the survey to rate the entire process. This 
can also be compared with staff ratings based on the data presented in Table 40. 
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Table 44 



Value of Process to Participants 
(by per cent) 



Very 

Helpful Helpf ul 



Not 
H elpful 



Staff Analysis 
Participant Survey 
Difference 



60. 3 



31.0 



8.6 



20.6 



58.8 



20.6 



-39.7 



+27.8 



+12.0 



The staff analysis represents the percentage of the 58 participants whose 
educational programs were deemed "effective", "moderately effective", or 
"ineffective" planning as a result of the entire process. The figures for the 
participants 1 evaluation were obtained from the survey form where physicians 
rated the value of the entire program on a scale of 1 to 9, with ratings of 
1-3 indicating degrees of "very helpful", 4-6 indicating "helpful", and 7-9 "not 
helpful". As in the previous table, a rating of 1 was considered 100% helpful 
ana a rating of 9 was considered 0% helpful. In spite of the differences in 
semantics, and the ways in which the numerical ratings were derived, the 
results are considered validly comparable. 

As is obvious, in spite of the support expressed for the separate procedures 
by the participants as presented in Table 43, the staff was generally more satis- 
fied with the end result than were the physicians. 

Re-testing 

As previously stated in this report, six of the original 37 participants were 
re-tested at the conclusion of Phase 1 of the study. The five who carried out 
educational programs showed statistically significant improvement in test scores; 
the one physician who did not carry out his educational program showed a de- 
crease. Due to the low number tested, the results were not considered signifi- 
cant, particularly due to inadequacies detected in the test bank. 



It was the intent of the;; staff to conduct similar re-testing at the conclusion 
of Phase 2. This was not done for two reasons: (1) time did not permit a sig- 
nificant number of participants to complete their educational programs, and (2) in 
the judgment of the staff, test results had come to be regarded as less than con- 
clusive evidence of the success or failure of the process, 
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To expand on the staff attitude toward testing, experience during Phase 2 
has led to a concensus that the degree of testing conducted does not render 
conclusive evidence. Much in the same manner that a physician uses diagnostic 
procedures to arrive at a diagnosis and regime of therapy, the test results are of 
most value when considered in relauon to other available data. If this analagy 
holds, then the degree of success or failure of the Individual Physician Profile 
process is best measured in the improvement in quality of patient care. 

Research in health care delivery -is just beginning to provide the methodology 
to make such a determination. Peer review, in a hospital setting, as demonstrated 
by Clement R. Brown, M. D. , at Chestnut Hill Hospital, PhiladelphiaVPennsyl- 
vannia; Robert L. Evans, M. D. , at York Hospital, York, Pennsylvannia; and 
Beverly C. Payne, M. D. , for the Michigan State Medical Society and Hawaii 
Medical Association, appears to offer one promising method. The staff is now 
exploring ways that this methodology could be applied to a physician's total 
practice, and as a consequence be used as an evaluation meohanisn for the 
process. 
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CONCLUSIONS 

The sole objective of the research was to develop a process by which an 
individual physician could identify and meet his personal educational needs, in 
relation to the health care he is called on to deliver. 

The intent was to define a physician's practice for him, test his instant 
recall of scientific knowledge in relation to that practice, provide a mechanism 
for interpretation of this objective data in the context of subjective information 
he provides, and provide specific information on which to base a continuing 
education plan. 

As such, it does have limitations. The practice data must be considered 
in its proper context: (1) it is a sampling and does not take into account 
seasonal changes in practice which may exist, and (2) it is dependent on accurate 
preliminary diagnosis. The testing data also have limitations: CI) most of the 
questions deal with instant recall, and do not deal with synthesis and application 
of the information involved, and (2) much of the testing is insufficient in depth to 
provide more than general guidance. The consulting procedure departs from an 
objective approach, and introduces intuitive judgments which may or may not be 
correct. \lso, while the educational plan is provided, there is no assurance 
that it is in fact carried out. 

Finally, and perhaps most important^ while the process does reflect on 
quality of patient care, it does not make an objective measurement of this. 

With these qusJifications, the process can be considered in its proper 
context, and conclusions drawn: 

General Conclusion s, 

I . While xh} r e are similarities among individual physicians* practices, they 
do vary substantially in terms of identifying individual educational needs . 

Prom the cumulative data nearly two-thirds of the diagnoses are in five 
ICDA categories, and generalizations can be made on this basis. However, there 
are variations when the specific diseases and conditions within these five cate- 
gories are considered. When the remaining one- third of practice, involving the 
ot^er 13 ICDA categories, is considered the variations in practice profiles 
become even greater. 
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In the context of this research, which is based on the health care which the 
physician is called on to deliver, the significance of these variations in practice 
profile are considereri sufficiently important to require definitive identification* 

2. It is possii>I° to identify individual educational needs for funnily practi - 
tioners . 

In phase 2 of the study, in the judgment of the pi , ject staff, design of 
educational programs was successful for 91.4% of the physicians who completed 
the consultation phase. The figure for general practitioners is 92. 2% and for 
specialists 85. 7%. 

However, the degree of indication given was much higher for the general 
practitioners, e.g. an educational diagnosis was more likely to be based on 
multiple factors rather than just one factor involved in the process. 

Consequently, while there is confidence that a genera) practitioner partici- 
pating in the program is likely to receive valuable guidance in planning his 
continuing education program, the data ndicate th. t a specialist is less likely 
to gain guidance of comparable value, 

3. It will be possible to identify individual educational needs for medical 
specialists . 

There appear to be two weaknesses in the present process which lead to 
limitations in identifying educational needs of specialists. The first involves the 
fact that the present test bank does not provide the discrimination required to 
identify potential areas of study. The second is that it does not lend itself to 
testing in medical areas, e.g. surgery, where techniques and procedures are 
an important factor. This second limitation, while primarily a reflection cn the 
test bank, also involves problems in the classification of patient data. 

However, there is confidence that these weaknesses can be overcome, and 
that the basic process is applicable to all medical specialties, 

4. Individual educational needs do vary . 

There is great variation in the educational programs proposed for the 
physicians who completed the entire process, and it wouJ appear that only 
limited reliance can be placed on any general needs identified. For example, 
Category 7, "Diseases of the Circulatory System" is most frequently prescribed 
as an area for study, and this is supported by the patient data, test results, and 
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determinations made by the participant and consultant. It is a proposed area of 
study for 62. 1 per cent of the 58 physicians who completed the process. 
Consequently, there is some basis for general educational programming in this 
area. However, the next most common area of educational need, Category 5, 
"Mental Disorders 1 ', ranks midway in the list of volumes of diagnoses and test 
results, but high in the priorities established by the consultant and participant. 
It is a prescribed area of study ..for 37.9 per cent of the 58 physicians. Thus 
identification of common educational needs is limited, and becomes substantially 
less useful as each additional category is considered. Equally, while it may be 
possible to generalize that a certain percentage of physicians have educational 
needs in a specific area of medicine, it still leaves the problem of identifying 
those physicians who do and eliminating those whc do not. 

5. It is possible to design personal educational programs for individual 
phy sicia ns, based on the health car i they are called on to deliver . 

Staff analysis indicated that educational programs were successfully designed 
for 91.4 per cent of those who completed the process; survey of participants, 
indicated that 85.3 per cent considered the entire process nelpful or very helpful. 

The role of the educational resource index was an important one in this 
design. Staff analysis indicates that use of the index or arrangement for a 
specific apprenticeship was possible in 83.5 per cent of the instances. Survey 
of the participants indicates they found the index useful 82.4 per cent of the time. 

Observations 

The study also revealed or confirmed certain information related to medical 
practice: 

!. Telephone communication between the physician and patient plays an 
important role in the practice of medicine . 

In the first study phase, 29. i per cent of patient contacts were by telephone; 
in the second study phase Lhe figure was 18.4 per cent. Utilizing either figure, 
the conclusion is that the telephone is an important factor in medical practice. 

It is surmised that some of the decrease between the two study phases is 
duo to loss of data, but tiiere is documentation that a substantial portion results 
from changes in telephone management within the office setting. 
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A standard part of the consultation procedure at the end of phase two was 
the provision of a telephone answering protocol. The data indicate a potential 
area of study concerning the role of telephone communications in medical practice, 
with particular attention to utilization of health manpower, 

2. Home visits are becoming an inconsequential factor in the medical 
practices 'studied . 

Home visits accounted for only 1.0 per cent of the patient contacts in phase 
two of the study. In a similar study in Massachusetts, they still constituted 
5.6 per cent of medical practice. The results in this study may reflect a 
regional situation related to Wisconsin, or perhaps only to the sample in the 
study. 

This may be an area of further study also, with attention to the types of 
diseases and conditions involved in house calls, and the influence of such factors 
as travel and implications on such matters as utilization of health manpower. 

3. Patient contacts frequently involve multiple diagnoses . 

With 14,507 patient contacts and 23,911 diagnoses recorded during phase 
two of the study there was an average of 1.65 diagnoses per patient. Secondary 
diagnoses ranged from one to six. 

Further analysis of these data is warranted, to make judgments on implica- 
tions as to the diagnostic process and training in diagnosis. 

4. Volumes of patient contacts appear to vary by such factors as age of 
physician, type of practice, etc . 

On the limited data available, it appears that the volume of patient contacts 
follows general patterns when the age of the physician, whether he is in solo, 
small group, or large group practice, and other factors are considered. 

Certain of this data warrants further study, with -attention to practice 
efficiency and utilization of health manpower. 

5 . Time does not appear to be a barrier to carrying out cont ir u ing educa - 
tion programs . 

Of 48 participant ; surveyed, 83. 3 per cent indicated they could leave their 
practices for a week or more at a time to carry out continuing education 
programs. 

O 7f 
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Since time is one of the factors most often listed as a barrier to continuing 
education, this observation warrants further investigation. The preponderance of 
group practitioners in this study probably had a major effect on the results. 

6. Physicians apparently know the methods by which they learn best . 

A corallary to designing a continuing education program involves efficient 
and effective utilization of the time devoted to carrying out that program. Some 
individuals, learn best in one method of study, others by another, some by 
multiple methods. 

Sometimes the educational task dictates the preferred educational method, 
e.g. learning a technique or procedure may be best done by doing, rather than 
reading or listening about it. Where there is an option, the physician's individual 
characteristics should be taken into account. Experience in the consultation 
process indicates that physicians can identify the methods by which they learn 
best; further study might be valuable to confirm these intuitive conclusions.. 

7. Procedures in the management of medical practice vary greatly . 

An important factor in a physician's efficiency is the way in which a 
physician manages his practice. The consultation process indicates that thus 
varies greatly among the participants. A factor which causes one physician 
great difficulty may be handled very efficiently by another. While an effort was 
made to discover and communicate effective practice management procedures 
during the study, there appears to be little effort to do this in a substantial way. 

Since there are major implications for effective utilization of health care 
personnel, rhe entire area of practice management appears to warrant study. 

Analysis of Procedures 

On the assumption that the results of the research appear to warrant: 
(1) expansion of the program to include substantial numbers of general practi- 
tioners in Wisconsin, (2) adoption by others in continuing medical education, and 
(3) continued improvement in the process and expansion to include other medical 
specialties, some conclusions related to the process are warranted. 

1. The resources developed appear adequate, but further effort is either 
required or desirable to accomplish the above. Some changes in the classification 
system are required to make the present procedure more effective. This includes 
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resolving the problems caused by categories 16 and 17, as previously explained. 
It also includes broadening the classification system to provide for the basic 
sciences and techniques and procedures. 

The test bank is now considered adequate for testing general practitioners, 
and it has particular strength in its relation to clinical practice. It is considered 
desirable to expand the volume and distribution of questions in the bank, and the 
conflict between the desire to test broadly on the practice profile but in 
significant depth in particular areas must be resolved. The advisability of 
weighting diagnoses so that those which are life threatening, or likely to result 
in disability, become more important in test composition must also be resolved. 
A challenge exists in developing the test bank to include the various intellectual 
processes involved in, medical decision making. At present the test bank primarily 
relates to instant recall of factual information. It is also possible to test in 
interpretation, application, analysis and synthesis. The development of the bank 
to cover these additional intellectual processes would be a substantial task. 

There are still some problems with the computer programs developed for 
the project. These include changes to make the programs more responsive to 
needs of the Wisconsin project, and documentation to permit them to be easily 
adapted to the needs and equipment capabilities of others interested in initiating 
similar projects based on the resources presently developed. 

2. Cost of the process, as with most research and development, has been 
substantial. Because of the large amount of staff time and data, manipulation, 
the on-going costs are also substantial. There is continued research and develop- 
ment, as outlined above, as well as maintenance of the resources such as the 
test bank and educational resource index. In addition to these, the base cost of 
performing the process for one physician has been documented to average 
$215.90. Some economies already appear possible. By eliminating the two 
visits to the physician, that of the project specialist to explain data collection 
and that of the consultant, the average individual cost is reduced to approximately 
$125. Both functions will be attempted by telephone. Othe~ possible economies 
will be explored, but there is danger in reducing effort and * to the point 
that the process is significantly less valuable to die physician, * balance must 
be achieved where die costs appear to equal the value received by the participant 
with consequent self-support. 
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3. V/hile the achievement of the research objective related to development 
and refinement of a process appear sufficiently conclusive for expansion of the 
concept, the evaluation is not yet considered adequate. Expansion, it is thought, 
can be justified on the basis that the process fills an existing void. Evaluation 
by participant reaction, staff analysis, and limited re-testing is not considered 
conclusive. 

The aim of the project staff is to attempt to evaluate the process in terms 
of improved patient care. While the details are not worked out, it would involve 
utilization review based on both office and hospital patient records, and a com- 
parison between delivery of health care prior to arid following completion of the 
educational program developed by the process. 

Long- Range Implications 

As indicated in this report, the process developed and data collected open 
up significant avenues of additional research, much of it unrelated to continuing 
medical education. 

The data gathering procedure, even with its limitations and poor sampling 
techniques, appears to have provided information not available elsewhere. With 
improved techniques it may render data critical to major decisions being made 
in the health care field today. 

Physicians can and will be tested on scientific knowledge. The present test 
bank may have limitations, but it appears to be u; ique in its relation to clinical 
practice. If the additional evaluation previously suggested establishes a relation- 
ship between test results and quality of deUvery of health care, the implications 
would be substantial. 

The subjective information gained 'through the consultation process has 
resulted in some insights into the clinical practice of medicine not previously 
available to the faculty. 

All of these provide a potential starting point which could have major 
implications on the continum of education in the health scir ":es. For physicians, 
this education is now fragmented into medical school, internship and residence, 
and continuing education. The process developed in this study, with further 
development, may provide the means by which data from clinical practice can be 
related back to the medical school curriculum and formal postgraduate training 
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period to identify strong points, weak points, and changes in clinical practice 
which should be immediately provided for by changes in curricula. Equally, 
there is a potential for information as to what types and numbers of allied health 
personnel should be trained to fulfill the needs of clinical practice. 
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Exhibit 1 
QUESTION EVALUATION SHEET 

Feel free to write anywhere on these pages 



Question number 
Reviewed by 



PART I: STRUCTURAL AND MEDICAL VALIDITY 

p.JK. NOT O.K. 

1. Question Statement: 



IF NOT O.K.. WHY? 



2 % Question Alternatives; 



3. Correct Answer; 



PART II: RELEVANCY TO CLINICAL SITUATIONS 

4. In your judgement , this question 
i6 most relevant to the following 
type or types of practice. (Please circle) 



G.P. Surgery 
Int. Med. Others 
Pediatrics 



5. The information presented in 
this question pertains to 



a) ; 



b) 



c) 



m common clinical 

situations and " on the 
spot 1 ' decisions 

decisions requiring 



commonly available 
diagnostic tools and 
t roceduras 

problem* or 



techniques requiring 
special knowledge or 
training 



6. The extent to which this item is 
characteristic of situations and 
problems faced by clinicians is: 



: 0 
NOT 

characteristic 



1 



4 5 
VERY 
characteristic 



1. COMMENTS AND FRUSTRATIONS: 
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Exhibit 2 



QUESTION REVIEW BY SUBJECT-MATTER EXPERT 

Date: Question Number: 

Reviewed by: 



; Circle one 

A. Is the answer to the question scientifically ( 

correct? « Yes No 

If Yes, is it the best possible alternative? Yes No 

If No, why not? 



B. Are the other options reasonable? Yes No 

If No, which ones are not and why not? 



C. Do you consider any part or parts of the 
question statement to be scientifically or 

medically erroneous? Yes No 



If Yes, please mark the part or parts 



Exhibit 3 



PHYSICIANS INFORMATION 

1. Name &l***> 

2. Specialty „_j^f"^£_ Second Specialty 

3. Office Address ~ 

4. Office Phone „„ZL 

5. Second Office Address 

6. Second Office Phone _ 

7. Home Address ,„__. ,___„„ 

8. Home Phone 

9. Office Contact __J^j_^ % _ _ Position jA^J*^£i^^ 

10. Physician's Age 

11 . Total years in practice __J1__ 

12. Years in present location S~ 

13. Size of Medical Community L-^-Ji 

14. Size of city ($) ^JJjj mm 

15. Office situation JjL) £t$ ± 

16. Type of Practice I^fT) . O rhb's - ITS Specs* t< ru 

17. Ancillary Workers - . 

RN 1 s j p$tT T***i Other supporting personnel /J[ 

18. Days off J^^^ 

19. Total hours per week working 
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Medical Education 

1. Medical School when ^ ^ 



2. Internship _____ When ^*__£_L 

3 • Residency _J^_ When _ h 



What Specialty ._„„_. .„. ,_ 

Where _ _ When 

Other Specialty 

4. Board Eligible _^ ^ 

5. Board Certification 

6. Society Memberships C^t^ Z^ J^u^/ > J*-t**fy 



Hospital Affiliations 

1. Primary hospital affiliation 

Where _ ._„„„„__„„_ . 

How far from office _ ^jt_ s$&r7^> .„„ 

2. Other hospital affiliations 

Where _____ „_„ 

How far from office 

3. Number of beds in primary hospital _______ 

4. Practice privileges in primary hospital : 

Medical Complete _ .„__ Limited 

Surtjicjl Complete ___________ Limited 

OH Complete _ Limited 

Other Complete Limited 



m^nitaJ. A JTi M ations (cent.) 

>. :>:.ecic.l I'aci J 1 1 itjii available (uny Hospital) 

.f^j-'orom.ry Care Moo-j»ut aJ hi\.oiu5i ve Care 1 

^ 1 at endive fare f^imern-Micy ! : ouni 

^elf-Caiv biiito _._Other 
: . ..uucatu'na] l-i.'sourceu Available 



To c' j c: 1 1 ! i i i r . C or i } o J -e not 
V>\.u IT Moo t .i n»^o 



f ionic a I Auoi U 



_ ^'io]0:'!iU!n: i'uiC I U Coil (\ 
■ . ".in* ;io Con »:oj ■ t Ki I 

S i.dal mc:co.;ii orociiuiv 
;-,i;ia:;:l.iun i' r*o;r I'cUii:; 

J v i c* you require all your ecu cut ion in your nomo community? 

if not, for how lonr, at one time can you "leave your practice? 
1- j nay a 



1 .CjIV tllul) i 1 Wt;Ouo 

in*'.; man;/ *;uvm times nor year coulii yuu J(, v avo your 1 practice? 
..iiuL oeucat.i ona.i resource:; <.o you u;*i«.* now vjouIcJ like to try 



.Uui Jo t'elicej 

S'i - t^rat;uate P re c«'i i tor::*.*! Lp 
V railu:e;»iiip 

Auuio Vj..*j ual ( fx In.. , tape;; , 
am; ;i Ji (i'Co ) 

I ) -ok, r amine u i n:; t rue t i on , ii oi;ie 
vitnuy course..; 

• ■■•ileal text., tutu joui'i;;i]:; 
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PHYSICIAN DATA 



a 


9 


10 


1 1 













M 


\ 5 


16 


17 


16 


19 


20 



















DAILY RECORD F 



DATE 



PT. ft 



AGE 



S A 
E C 
E 



CATEGORY NOS. 



SIGNIFICANT 
SIGNS & SYMPTOMS 



. 2 1 


22 




^' i 

4} 25 


26 


27 


2B 


29 


30 


3 : 


32 


33 


34 


35 


36 


37 


38 


39 


40 


4 1 


42 


43 


•4 4 


45 


46 


47 


48 


49 


50 


5 1 


52 


53 


54 


55 


56 


57 


53 


59 


60 
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23 


24 


2 5 


26 


28 


29 


30 


32 


33 


34 


35 


37 


38 


40 


4 1 


42 


43 


44 


45 


47 


4fl 


49 


50 


51 


52 


54 


55 


56 


57 


aa 


59 
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LY RECORD FORM 



SIGNIFICANT 
SIGNS & SYMPTOMS 



MAJOR 
DIAGNOSIS 



1 Specialty 

2-6 Physician No. 

8-9 Age 

10-11 Years in Practice 



CONTRIBUTING 
DIAGNOSES 



13 Medical Ed. 

14 Size of Medical Community 

15 Type of Practice 

16 Size of Patient Community 

TESTS 
ORDERED 



17 Type of Community 

18 Postgraduate Training 
19 

20 

TREATMENT & 
DISPOSITION 



1 \ 59 
t 


60 




_ 


















i 





_ 
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i 
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\ 

1 
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i 
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Exhibit 5 



Prn s rcfTvN — ^2S t est" 



lfVHEN lichee planus occurs on the o*al mucosa it is often confused 
with: 

a» ca*ct"*omv~ '* 

B • leukqplak 1 a. 

C- TUBERCULOSIS, 

□•GINGIVITIS. 

E* FOROYCE'S disease. 



2 • vV H t C H ONE OF THE FOLLOWING STATEMENTS CONCERNING THE TREATMENT OF 
INFECTIOUS MONONUCLEOSIS I 5 CORRECT: 

A • PENICILLIN is CONTRAINDICATED BECAUSE OF THE INCREASED TENDENCY 

TOWAKO SENSITIZATION IN THIS DISEASE. 
B« ALL CASES OF THROMBOCYTOPENIC PURPURA AHICM HAVE BEEN OBSERVED 
... ...„ t £ijt\ tv^ I c A T I N o~ TTTTS^'D I S E A 5 E ~TT ft"V E~ -S ET^PT TFT E K U5E OF ONE SPECIFIC 

ANTIBIOTIC. 

C- IMMUNE SERUM GLOBULIN IS KNO*VN TO DECREASE THE OUR A T 1 ON —OF THE 
iLLNC'jS. 

D« TETRACYcLiNt IS APT TO INCREASE THE CHANCES OF DEVELOPING 
HEP AT I T I S • 

-E> ' TN"T" rtTt*OT7T'S" irATVTT "N OT*~ N OTrCE3T5*L: v L'THl T E~I5 T'FTE ^TRXTT0^~~"0T r * ThIT 
■ DISEASE. 

i 

j.at w h at le v el of urinary output is the ...condi t ion defineo as jligurm; 

A • NO URINARY OUTPUT 

. , B * " LESST«Arj-Tri0 ML D'AlLY ~ ~ " 

."C"» LF.SS I HAN 2 U[] ML DAIlY 

!)• LESS I Han 300 M L ()AIuY ' 
. . LESS THAN HQo H L i\AI|.Y . 

HARVEY, A • MC GEhEEi t D • i THE PRINCIPLES &' PRACTICE OF MEDICINE, 
A P P L E T 0 N - CT. TIT UTTY^CR OF TS J I 7TTT En I T TON 7 "T^WJ PVrj'ff " 

H. PERIORBITAL EDEMA MAY PRESENT " AS THE INITIAL SYMPTOM IN ALL UUT WHICH 
ONE OF THE FOLLOWING DISEASES: — 
A • INSECT B I TES 

Bi - "URTTCA-RTA )M:Tr,j~~ *~ — 

C* TR I CHINOS IS 
. {)• GL°ME K Uc DNEPhR I T IS 

e • nePhr* 1 ^ .;:*osis 

HARRISON, T.R.i f.O«i PRINCIPLES OF 'INTERNAL MEDICMEi & T H EO., 
q |^:<S**tt H+ttri — rVjr&T -P"*T9 : ~ ~ • 
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PHYSICIAN 22SS2 



an eleven-year-old gh*l has had attacks of epigastric pain for one 

Yt.AR. HER PAKKMTs ARE V£rY rtORKlEO, ANQ RECENTLY THE PAIN HAS MADE* I T 
MKCESSAKY FOk H F.R 10 WITHDRAW FROM SCHOOL. CAREFUL AND REPEATED 
D l AGNDSTl C STUD 1^5 "HAVE ~R E V E A CE 1) "~N 0" "0 RU A NTcT 01 5'EXSE" • NX3f"~ 61? 0 S 5 "~ 
P^YCHOPaTHOLOG Y j5 APPARENT IN THE PATIENT OR OTHER FAMILY MEMBERS. 
THE NEXT STEP SHOULD PROBABLY BE! 

A» INTENSIVE DRUG THERAPY U5JNG AN ANTISPASMODIC AND A TRANQUILIZER 
*BV PSYCHOANALYSIS FOR THE PATIENT, IF HER PARENTS CAN- AFFORD I T 
C« EXPLORATORY LAPAROTOMY 

D • ■ STKIC^ RULES FOR MANAGING E A C H " a T T7TC K* "WH I CK M i NTM'I IE UPROAR ANT> 
PARENTAL PARTICIPATION - PLUS IMMEDIATE RETURN TO SCHOOL 
Ei PSYCHOTHERAPY INVOLVING THE ENTIRE FAMILY GROUP FOR A PROLONGED 
PERIOD ?~ 



PROPHYLACTIC" ANTIBIOTICS 'A'RE OF PROVEN' VAC^T TN ! ~ 
A» MEASLES (RIJHCLLA) 
(*• RESPIRATORY DISEASE 
C» RHEUMATIC FEVER 
Of GLOME K ULONEPHR I T I S 

HARBISON-, iiK, ,E0. i PRINCIPLES Or I NTER*tttU ~HFD FCTNE f 5TH~XTJT f ~ 

M.C G RAW H ILL, 1966, P. 1MB? 



A MlRLY TYPica'l PATTERN OF CONCENTRATIONS OF ELECTROLYTES IN A 



PAT 1 ENT 


WITH 


RESP I RAT'ORY 


acidosis is; 






N a 






CI 


"TOTAL "CO"? 


" " PH" 


Ai 1 1 7 


M E Q / l 


l .a HEQ/L 


86 MEU/l 


5MM/L 


7.29 


B» t 27 


M E Q / l 


3t8 M E 0/ L 


9 0 ME'^/L 


20MM/L 


7.09 


C - 3 37 


M t Q / L 


4.5 MEQ/L 


90 HEQ/L 


*40MM/L 


7.31 


D» ' I,H2 


M MJ / L 


H.8 MeO/L 


93 MEU/L 


28MM/L 


7.3H 


E ♦ 1 6U 


mCu/l 


2.8 M£0/l 


! I 7 MEW'/L 


1 5MM/L 


7.22 



T Hh COMMONEST UNDERLYING CAUSE OF VOLKMANN'S ISCHEMIC CONTRACTURE IS ! 

A* LACERATION OF THE MUSCULO T ANE OUS NERVE 

B» LACERATION OF THE RADIAL NERVE 

C» CONSTRICTION OF ThE CIRCULATION BY A TIGHT CAST 

D» CONTRACTURE OF THE MUSCLE SURKOUNOTNG THE FRACTURED" BONE "' 
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9. A YEAR OLD TRUCK DRIVER WaS In AN ACCIDENT AND SUSTAINED A SEVERE 

LACERATION OF ONE LEG, HE WaS TAKEN TO A NEARRY EMERGENCY ROOM WHERE 
THE WOUND toAS CLEANED AND SUTURED • THE PATiENT RECEIVED TETANUS 
ANTrSENUM" '(HORSETt TWO WEEKS LATE*~i "THE" P A"T I E U T " B E<aKW' Tl) 'TTE Vlf OTP" r P71TN 
IN THE RI^HT SHOULDER. ThEPAIN F.XTenqED To HIS H A NO • TWO DAYS LATER 

•m.u days Following the accident) he developed a similar though. less 
sensitive area on the back of his right hand. three days .later [19 
days following the accent) the p atient began to not i ce weakness in 
elevation of his right arm a n d weakness of grip in the right h and • 

' WH O F " T KE ' F 0 \ L 0 C, I S ~*T HE ' M O ST \ Pk E L"Y "67 A G N 0 s1~S V :' : ~~ 

A* AMYOTROPHIC LATERAL SCLEROSIS 

B POST- ] NFEClTToN POLYNEURITIS 

C* SERUM NEURITIS 

..*[)• CONVERSION REACT 1 ON ( HYSTER I A ) 



OmVHICH. ONE OF THE FOLLOWING STATEMENTS -ABOUT DISSEMINATED HISTOPLAS" 
^MUSi s I s CORRECT? 

A* BLOOD CULTURES ARE POSITIVE I LESS T H AN S* OF ALL PATIENTS, 
B» THE ADRENALS ARE ALMOST INVARIABLY UNINVOLVED. 
[)• PLEURAL EFFUSION IS EXTREMELY COMMON. 

— E-r- ~chfs T in v ox V e m;e N T " T*s^PRT:"STr^r-i ^~iriirj^zr^w%^ 



{ \ll of the following are common si^ns df cardiac failure in infants 
except: 

a • . c ard i omegal y • 

b • tachycardia/' v ' ' ~ ~ " ~ "~ " 

• c • hepatomegaly . 

()• pulmonary Rales. 

E • RESPIRATORY oISTRESS. . 



2 • A C h t co FD UNO "TO "VTA V E HYP 0*G^A M"M ATi CO B U L I N rwi JT'Tn D'TTM PI-RTF EN fA^D E VEX' 6 P S~ ~ 
s MARKED LYMPHEDEMA. WHICH OF THE FOLLOWING IS THE MOST LIKELY DIAGNOSIS 
A* SWISS * Y P E AGAMMAGLOBULINEMIA 
B« VARIANT OF THYMIC APLASIA. 
C» WHOOPING COUGH 

D» INTESTINAL LYMPHANGIECTASIA 
C*" A"CUT'C~~L YftPVTA T I C"X EUKEM I A " 



LEFT SIDE OF ANSWER SHEET 



RIGHT SIDE OF ANSWER SHEET 
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SENT 


TO M.D. 


AFTER 




PAINTING 


jestion 
No. 


Acc. No. 


Cat* no. 




Ans « 


Quest 




A 


B 


C 


D 


E 


F 


1 


4710 


AA * COO 

09A528 


2 


B 


1 


B 


B 


B 


B 


B 


B 


2 


4739 


0LH075 


1 


E 


2 


E 


E 


E 


E 


E 


E 


■ 3 


2959 


L6A786 


L 


E 


3 


E 


E 


E 


E 


E 


E 


4 


1820 


17R989 


0 
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4 


E 


E 


E 
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E 


E 


5 . 


3369 


16A7B4 


1 


D 


5 


D 


D 


D 


D 


D 


D 


6 


3995 


08A465 
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C * \ 
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C 
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C 


C 


C 


7 


4733 


L8A003 
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8 


2828 


L7B813 
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C 
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c 
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9 


3997 


L7J89L 
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C 


c 


G 


10 


4840 


16A783 


2 


E 


10 


E 


E 


E 


E 


E 


E 


11 


4558 


L6A788 


1 


D ; 


LL 
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D 
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12 


0170 


04A289 


2 


d ; 


12 


D 


D 


D 


D 


D 


D 


13 


5107 


L6A787 


L 


A 


13 


A 


A 


A 


A 


A 


A 


14 


4463 


L7R989 


0 


D 


14 


D 


D 


D 


D 


D 


D 


15 


3763 


17H873 


0 


B 


15 


B 


B 


B 


B 


B 


B 


16 


1807 


12A686 
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B 


16 


B 
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B 


B 


B 


B 


17 


2896 


17R989 
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17 


C 
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C 


G 


C 


C 


18 


3875 


17B816 


0 


D 


18 


D 


D 


D 


D 


D 


D ■ 


19 


3962 


08C486 


1 


C 


19 


C 


C 


C 


C 


C 


C 


20 


4635 


08A462 


1 


D 


20 


D 


D 


D 


D 


D 


D 


21 


4895 


16A738 


2 


D 


21 


D 


D 


D 


D 


D 


D 


22 


5052 


18KL06 


1 


B 


22 


B 


B 


B 


B 


B 


B 


23 


3508 


. 17B813 


L 


C 


23 


C 


C 


C 


C 


C 


C 


24 


4061 


u.«.nu / d 


i 
i 






L 


C 


C 


C 


C 


C 


2: 


3243 


18B010 


L 


c 


25 


C 


C 


C 


c 


C* 


c 


26 


4559 


16A788 


2 


D 


26 


D 


D 


D 


D 


D 


D 


27 


1708 


09B532 


0 


E 


27 


E 


E 


E 


E 


E 


E 


RETAINED 


BY OFFICE FOR 


ITEM OPTION 


ANALYSIS 




RETURNED 
MARKING 


TO OFFICE 
PURPOSES 


FOR 
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Exhibit 7 



RESULTS OF CONSULTATION 

I EXAM SUGGESTS NEED IN: 
a) 
b) 

°) 
d) 

II YOU SUGGESTED NEEDS IN: 
a) 
b) 

c) :• 
d) 

III PRACTICE PROBLEMS FUSING DISTRESS: 
a) 

•b) ! 
O 

iy PATIENT PROBLEMS CAUSING DISTRESS: 
a) • • 

b) 
c) 

V FUN PART OF PRACTICE: . 
a) 
b) 

c) ■ 
d) 
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IMP LEMENTATION OF YOUR CONTINUING EDUCATION 



I EDUCATIONAL SUGGESTIONS: 

A Scientific 

1. 
2. 

3. 

• 4. 

- 5. . _ 

II YOU LEARN BEST BY: 

Audio 

Visual _ 

By doing 

Other _____ 



III YOU CAN LEAVE YOUR PRACTICE: 

• days at a time 
. times a year 
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INNOVATIONS IN PRACTICE 
I MANAGEMENT OF INCOMING TELEPHONE CALLS: 
A. 

;* b. 

c. . 

II DELEGATION OF RESPONSIBILITY TO ANCILLARY HELP 

A.. , 

b. ; 

c. , , 

III EDUCATION OF CLINETELE 

A. Brochures 

B. Films 

C. Slides 

■ ? 

D. By ancillary help 

E. Other 



IV PROBLEM ORIENTED RECORDS 

1. Weed, L. : "Medical Records, Medical Educa- 
• tion and. Patient Care", "1969 Press of Case 

Western Reserve University, Cleveland, Ohio 
• MlC-6 * 

2. Bjorn, J.C. and Cross /H.D.. : "Problem - 
oriented Practice": Modern Hospi tjal—Rress . 
Chicago, Illinois-, McGraw-Hill Publications 

• ; co. u* *.j 
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CHARACTERISTICS OF PRACTICE 



- 1970 - 4days 

September 4, 8 r 16, 21 



I Total patient contacts for 4 days 145 



II Type of patient contact 
office 
, telephone 
hospital 
house calls 
emergency 
III Character of practice 
Male 52.3 % 
Female 47.6 % 



av. 21 pts/day 
av. 10 pts/day 
av. 3 pts/day 

av. pts/day 

1+ pts/day 



IV Age Range 



Office 



Under 14 
Under '10 98 

40-6'j 

65 & older 



76.3 % 

% 
% 



Hospital 
64 % 



92 



to 



125. 



22552 

TEST RESULTS 

1. Percentage of practice covered by test — 83% 

2. Total number of questions in tett 

3. Number correct 

4. Number wrong 

5. Percentage correct 

6. Total number of diagnoses 



97 



28 



77.6% 



250 



Results by Category: 



Cat. # 


# of Dx. 


% of Pr 


18 


66 


26.4% 


17 


42 


16.8 


16 


24 


9.6 


8 


18 


7.2 


9 


16 


6.4 


1 


13 


5.2 


12 


10 


4.0 


6 


9 


3.6 


4 


8 


3.2 


15 


7 


2.8 


5 


5 


2.0 


10 


5 


2.0 


7 


2 


.8% 


11 


0 





24 
17 
26 
9 
7 
9 
6 
2 
3 
4 
4 
5 
1 
1 



17 


7 


70.8% 


15 


2 


88.0% 


18 


8 


69.0% 


8 


1 


88.8% 


7 


0 


100.0% 


.9 


0 


100.0% 


2 


4 


33.0% 


0 


2 


0% 


2 


1 


67.0% 


2 


2 


50.0% 


3 


1 


75,0% 


5 


0 


100.0% 


1 


0 


100.0% 


1 


0 


100.0% 



o 
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Results by Sub-Sub Category 



t; 

Cat. 
Number 



18 



Sub-sub 


# of 


% Of 


Total 




1 




Category 


Diagnoses 


Practice 


Items 


Correct 


Wrong 


Correct 


Immun & Sensitivity 


28 




6 


4 


2 


67 


Lab . Exam 


7 




5 


3 


2 


60 


Innoc & Vacc. 


5 




5 


3 


2 


60 


Well Baby 


21 




5 


4 


1 


80 


39 _ . 
3ft Postop 


5 




3 


3 


0 


100 



17 



873 Laceration Head 


2 




1 


1 


0 


100 


884 Mult. " Upper Limb 


0 




1 


1 


0 


100 


907 Mult. " Unspec. 


0 




: l 




0 


100 


810 Fx. Clavicle 


2 


• 


l 


1 


0 


100 


813 Fx, Radius & Ulna 


2 


5 


4 


1 


80 


816 Fx. Phalanges Hand 


4 




1 


1 


0 


100 


807 Fx. Ribs, sternum & 


0 




1 


1 


0 


100 


Larynx 


0 












315 Fx. Metacarpal Bones- 




1 


0 


1 


0 


960-39 Adverse Drug Effects 


5 




4 


4 


0 


100 


930 Foreign Body Eye 


1 




1 


1 


0 


100 



16 



780 Sx, C.N.S. & Spec. Senses 

n 82 Sx. Cardu. & Lymphatic 

* 783 Sx. Resp. System 

7j}4 Sx. Upper GI 

7 85 Sx. Abdomen & Lower GI 

78 6 Sx. GU System 

7 87 Sx Limbs & Joints 

788 Gen. Sx. 

78 9 \bnorm. Urin. Constit. 



1 


2 


1 


1 


50 


1 


2 


2 


0 


100 


3 


3 


2 


1 


67 


2 


2 


2 


0 


100 


3 


3 


3 


0 


100 


2 


2 


1 


1 


50 


3 


3 


3 


0 


100 


8 


6 


2 


4 


33 


1 


3 


2 


1 


67 



465 URI 3 

486 Pneumonia, Unspec. 1 

461 Sinusitis 0 

480 Uiral Pneumonia 0 

4 90-9 3 Bronch, Emph , Asthma 1 



540-69 Dis Intestine & 8 

Peritoneum 
520-29 Dis. oral Cavity, Sal. 5 

gl. Jaws 
530-37 Dis. Esoph, Stom, 2 

Duodinum 
570-77 Dis. Liver, Galbl., 1 

Pancreas 



3 
3 
1 
1 
1 



1 
2 
3 
1 



3 
2 
1 
1 
1 



1 
2 
3 
1 



0 
1 
0 
0 
0 



100 
67 
100 
100 

ion 



0 .10 0 

p 100 

0 100 

0 100 
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Cat. Sub-sub # of 

Num K ^r Category Diagnoses 

\ 


% of 
Practice 


Total 
Items 


#- 

Correct 


# 

Wronq 


Correct 


009 Diarrheal Dis 
034 Strept Throat 
075 Infect. Mono 
1 079 Other Viral Dis 
017 TB other organs 
032 Diphtheria 


4 

2 

* 

3 
2 
0 

0 




2 
1 
3 
1 
1 
1 


2 
1 

. 3 
1 
1 
1 


0 
0 
0 
0 
0 
0 


ion 

1.00 

l on 

.100 
100 
100 


692 Other Exzema & 
Dermatitis 
^2 708 Urtichria 

691 Infantile Exzema & 
Relat. 

709 Other Dia. Skin 


4 

1 
0 

0 




3 

1 
1 

1 


1 

0 
0 

1 


• 

2 

1 
1 

0 


33 

0 
0 

100 


6 320-58 Dis Nervous Sys. 


1 




2 


0 


2 


0 


- Dis. Bid & Bid. forming 
Org . 


• 9 


- 


3 


2 


1 


67 


15 Perinatal Morbid & Mortality 


7 




4 


2 / 


2 


50 


.| .ental Disorders 


5 




4 


3 


1 


75 


10 580-99 Dis.Uriftary System 


5 




5 


5 


0 


100 


7 390-98 Rheum. Fever & 
Heart Disease 


0 




1 


1 


0 


* 1-00 


11 Compl . Preg & Puerperium 


0 




1 


1 


0 


100 
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RESULTS OF CONSULTATION 

I EXAM SUGGESTS NEED IN: 

a) Other examinations without Illness 

b) Diseases of Skin and Subcutaneous Tissue 

c) Symptoms and ill-defined conditions 

d) Fractures, Dislocations, Sprains and Strains 

II YOU SUGGESTED NEEDS IN: 

a) 
b) 

d) 

III PRACTICE PROBLEMS CAUSING DISTRESS: 

a) Phones 
b) 

IV PATIENT PROBLEMS CAUSING DISTRESS: 

a) Neurological and Emotional 

b) 

c) 

V FUN PART OF PRACTICE: 

a) Newborn 

b) Well baby 

c) Cardiology 
d) 
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INNOVATIONS IN PRACTICE 



I MANAGEMENT OF INCOMING TELEPHONE CALLS: 

A. Keep tally of reasons for calls. 

B. Procikflure gruide for calls could be set up. 
C. 

II DELEGATION OF RESPONSIBILITY TO ANCILLARY HELP 

A. Take inventory of tasks performed to see what can be 
q 9 deleaated - Set up a procedure guide to avoid legal 
entanglement 

C. 

Ill EDUCATION OF CLINETELE 

A. Brochures x 

B. Films X 

C. Slides 

D. By ancillary help x 

E. Other 



IV PROBLEM ORIENTED RECORD:; 

1. Weed, L. : "Medical Records, Medical Educa- 
tion and Patient Care" , 1969 Press of Case- 
Western Reserve University, Cleveland, Ohio 

2. BJorn, J.C. and Cross, H.D.: "Problem - 
oriented Practice": Modern Hospital Press, 
Chicago, Illinois, McGraw-Hill Publications 
Co. 



IMPLEMENTAT ION OF YOUH CONTINU I NG EDUCATION 
I EDUCATIONAL SUGGESTIONS: 
A Scientific 

1. General well-baby and. Child Care annual confer 

2. General annual meetings for Internal Medicine 
3. 

4. 
5. 

II YOU LEARN BEST BY: 

Audio 

Visual 

By doing X 
Other 



III YOU CAN LEAVE YOUK PRACTICE: 

14 days at a time 
1 times a year 



THE ED UCATIONAL RESOURCE INDEX 

The Educational Resource Index is a compilation of educa- 
tional opportunities available to the t practicing physician. 
Information is included on: 

A, Away 

1. Conferences 

2. Post-graduate Traineeships (or appren- 
ticeships ) 

B. Home 

1 • Audiovisual Material (tapes , s lides , 
f i 1ms ) 

2. Programmed Instruction (and home study 
courses) 

3. Medical Texts and Journals 

The pertinent information on the above offerings is coded 
into one or more of the appropriate disease Categories 
listed on the enclosed request form. 

The included computer print-out covers the categories of 
the Resource Index which have been suggested to you by . 
your educational consultant. The blue dot marked programs 
have been specifically recommended for your needs or are 
felt to be particularly outstanding oy the consultant. 
If you feel thez you would want additional categories of 
information please fill out and return the Educational 
Resource Index Request Form. Check the boxes tiiat conform 
to the type of instruction and Disease Categories desired*. 
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KEY to educational resource mm* 



ENTRY. DATE : 
DELETE DATE 

99/99/99 - 
38/88/68 = 



POST GRAD. 

CONE. TRAINEE. 

Beginning date 
: Ending date of 

program 
DATES UNKNOWN 
DATES TO BE ARRANGED 



AUDIO- PROGRAMMED MED. TEXTS 
VISUAL INST. & JOURNALS 
Date produced or published 
Not Meaningful 



KEY TO ABB REVIATIONS 
FOR: G - General Practioners 
S - Specialists 
B - Both 

METHODS: AV - Audiovisual Aids 
BR - Bedside Rounds 
Clin C - Clinical Conf. 
LC - Live Clinic 
Lab - Laboratory Work 
Lec - Lecture 
0 - Open Question Periods 
OP - Enrollee Observes Procedure 
PD - Patient Demonstration 
Pan - : Panel Discussion 
PP - Enrollee Performs Procedure 
Sem - Seminar 
R- Radio or Telephone 
Surg - Operative Human Surgery 
TV - Television 
PI - Programmed Instruction 



KEY TO ADDRESSES 

Albany Dept. of PG Medicine Tapes , 47 New Scotland Avenue 
Albany, New York 12208 

Audio- Digest Tape F oundation , 1250 South Glendale Avenue 
Glendale, California 91205 

Wisconsin Dial Access Tapes , In Madison: 262-^515, Other 
areas : 1-800-362-817^ 
Wisconsin Single Concept Films 

Wisconsin Telephone/Radio Conferences , 61 4 Walnut Avenue 
gpj£ Madison, Wisconsin 53706* 




t L 3 *ElL UAJY AND CHILD CaRE 
• GET ,UOCTOW f W £LL H A B Y AND CHILD CARfc 
JCTOH 

WELL WABY ANo CHIlH CAKE 
A*AY 

CONFERENCES 



1 1 234 



ENTRY DATE OH/ 12/7 1 DELET^ DATE QH/U/71 

PEDIATRIC ENDOCRINOLOGY i G^OWTHi */ METaBOiJSM 

SPONSOR- JOHNS HOPKINS UN1V SCH OF MED , 601 N BROADWAY, 

bALTlfiUREi MARYLAND 2120b 
LOCATION- SAME ADDRESS 

FEE- 5 1 75- FOR- B 5 LIMIT 120 

LENGTH- 5 DAYS, 3b MRS, 

MLTH0D5- AV, CLIN Ci Op, LLC t LC i 0, Pa'^i P !> * SEM 



117 17 



ENTRY DATE OH/12/71 DELFiE DATE OH/17/71 

SPECIALTY REVIEW COURSE lN PEDIATRICS 

SPONSOR- COOK COUNT Y GRADUATE SCSuOL OF MEDICINE 

707 s« kood STt, Chicago, ill, 606i2 

FOR- PEO t 

LENGTH- b 1/2 DAYS 



10977 



11226 



1 1 27H 



11271 



ENTRY D A * E Oh/IS/71 DELETE DATE OH/15/71 

1 2 TH ANNUAL MATERNAL 6 CH [ L D HE A L T H I N ST I 7 0 1 E 
SPONSOR- PENNSYLVANIA mED SqCIETY 

BYPASS & EKFOMD RO., LEMOYNE, PENNSYLVANIA 170H3 
LOCATION- HE/SHE Y MOTOr LODGE 

CHOCOLATE AVE*, HERSHEYi PENNSYLVANIA 17033 
Ft E - £10 FOR- B, LIMIT 200 

LENGTH- 1 DaY,.H"'HRS 
METHODS- AV f LEC , 0 , PaN 

ENTRY DATE Oh/19771 DELETE DATE 04/2H/71 

SPECIALTY REVIEW COURSE IN PEDIATRICS 

SPONSOR- COOK COUNTY GraD ScH OF fitOt 

71)7 S rtO'OO ST t CHICAGO , ILLINOIS 606; 2 
LOCATION- SAME ADDRESS 

FEE- S1SU " FOR- S, LIMIT 85 

LENGTH- b 1/2 DAYS, 38 HRS« 
METHODS- AV , LEC , 0 

COMMENTS- INFORMATION & APPLICATION FORM AVAILABLE 

ENTRY DATE Oh/22/71 DELETE DATE 0H/2H/71 

PARANATAL RADIOLOGY 

SPONSOR- UNIV OF WIS, 307 N CHARTER ST t MADISON, AlS S3706 
LOCATION- .VIS C N T R i 702 XANqDON ST, M Ao I 5 0N S3 7 06 
FEE- 160 ' FOR- jJ i LIMIT 60 

LENGTH- 2 DA YS 

METHODS- AV, LEC, LC , Oi PAN 

ENTRY DATE 0*4/23/71 DELETE 0 ATE OH/23/71 

PEDIATRIC DAY:SUTT0N LECTURESHIP 

SPONSOR- MED COLL OF Va» HEALTH SCIENCE* OIV» VA 
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\\2 H 1 



COMMONWEALTH uNIVi BOX 9 I B RJc^^ONDi VIRGINIA 23219 
LOCATION* SAME ADDRESS . 
FEE- FOR- B 

LENGTH- 1 OAY t 6 HK5« 
METHODS- AV, LEC 

ENTRY DATE 0»*/26/7l DELETE DATE OH/30/71 

PEDI ATRIC R.ADIOLQGY 

SPONSOR- UNIV OF HO-COlUMBU MEO CNTR 6 EXTENSION dLDG, 

UD7 STADIUM HQ , COLUMBIA, MISSOURI 65201 
LOCATION- CHILDREN'S MfRCY HOSP, KANSAS CITY 

FEE- $17b FOR- S, LIMIT 15 

LENGTH- & DAYS, HO HRS # 

METHODS- AV, BR i CLIN C i OP. PP, LA8i L^i LC , 0, SURG, PANi 
- Po, SEM 

COMMENTS- BROCHURE. ANQ APPLICATION FORM AVAILABLE 



! 126B 



1 1003 



1 1 2 I 5 



IU2V8 



ENTRY, DATE 05/01/71. DELETE DATE 0&/U|/71 
POSTGRADUATE PEDIATRIC CONFERENCE 

SPONSOR- SCOTT b rfH 1 TE MEM HOSP & SCOTT* SHERWOOD t> 

BRlNDLEY FOUndi 2H01 S 3lSTi TEMPLE, TEXAS 76501 
LOCATION- SAME ADDRESS 

FEE- $10 FOR- B i LIMIT 150 

LENGTH- 1 DAY a bHftb« 

ME f HODS- AV, LEC, 0, P A N, bEM 

/ 

ENTRY DATE 0fiV06/7lV. DELETE DATE 0S/U7/71 

OPHTHALMOLOGY FOR THE INTERNIST, PEDIATRICIAN, AND GENERAL 
PRACTITIONER , 

„ SP0 _ N A°. R r^ AKVA . RC) S .^K« & tpT 0F C'OnT E» U C, 25 SHaTTUcK ST, 

" B OS TON , M A S S Q 2 1 1 5 
LOCAT I ON-MASS EYE AND E A & INFIRMARY, 2**3 CHARLES ST, BOSTON 
MASS 021 14 

F£Z-Sl$ FOR-B, LIMIT-100 

lEngTh-2 days, 12 hours 
methods-av , lec 1 pd 

entry date u5/o6/7i delete date u5/u8/71 
advances in pediatrics 

sponsor- univ of calif scm of med, med cntr, 

3rd 6 parnassys aves, san francisco, callp 94122 
location- same address 

FEE- V OR- B 

LENGTH- 3 DAYS, 16 HRS # 

METHODS- AV, CLIN C, LEC 0, HAN, S£M 

ENTRY DATE 0 5 / i CI / 7 2 DELETE DATE 0&/M/71 

ADOLESCENT MEDICINE 1 

SPONSOR-" HARVARD "MED ScHOOL, DEPT. OF c 0N T# EqUC« 

2b SHATTUCK STREET, BOSTON, Ma* s * 02115 

LOCATION- CHlLDREN"S HoSPt MEDICAL CENTER, BOSTON 
FEE- &175.00 FOR- B ,* LIMIT; 100 

LENGTH- 5 DAYS 1 32-36 HOURS 

METHODS- AV, LEC, 0, PaN,' Pd 
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11722 



ULLETt DATt Ob/lM/71 



ENTRY D AT L 1)5/10/71 
AUOLLSCtNT MLD1C1NE I 

SPONSOR- HARVARD MEUICaU SCHOOLi DEPT. OF CONTINUING 

2S SHAlTUCK St*» boSTON, MASS U2115 
LOCATION- CHILDREN'S HoSPHaL MEDICAL c tNT ^ H 
300 LONbWOOo AVE* | bOSTON 

FOR* P£D, 

LENGTH- 5 DAYS 



E0UCAT10 



1G359 



ENTRY DATE 05/12/71 DELETE DATE 05/13/71 

SIXTH ANNUAL INDIANA MULT I D I 5C I PL I NAR Y CHJLD CA«E CONFERENCE 
SPONSOR- INDIANA UNIVERSITY, SCHOOL OF MEDICINE 

1100 W. MICHIGAN ST#, 1 NO 1 AN ApOL 1 S , I NO • H6202 
LOCATION- SAME ADDRESS 

FOR- G ( LIMIT HO/ 
DISCUSSION GROUP 

LENGTH- 2 DAYS 

COMMENTS- INFORMATION AVAILABLE 



1 1 2S7 



ENTRY DATE U&/12/71 DELETL DATE 05/14/71 

INFECTIOUS DISEASES 

SPONSOR- AMEN ACAD OF PEDIATRICS, 1 BO 1 HlNMAN AVE, 

EVANSTON i ILL 602U4 
LOCATION- UNIV OF OKLA MED CNT« P 0KLAH0 ma ClTy 

FEE- S/S/llEMBLRi & 1 0 5 / n ON ME Mb E R FOR- S, LIMIT 100 

LENGTH- 3 D A Y S i 2H h^S, 

METHODS- AV, CLIN Ci LeCi LCi 0, PAN, s^N 



1 12bo 



1 1237 



ENTRY DATE Oft/12/71 DELET^ DATE 0&/1H//I 

INFECTIOUS DISEASES AND IMMUNOLOGY 

SPONSOR- UNIV OF OK L A mED CnTR, 80U NE 13TH ST, 

Oklahoma c i t y ( Oklahoma /3 i 

LOCATION- SAME ADDRESS 

FEE- FOR- 8 

LENGTH- 3 UAYS, 21 mKS. 
METHODS- AV , LtC , 0 , PaN 

ENTRY DATE 05/17/71 OELET DATE 05/21 71 

PEDIATRICS- I 

SPONSOR- HARVARD MED ScH, DEPT OF C0NT EDUC, 

25 SHATTUCK STi boston, MASSac hu SETTs 02115 
LOCATION- MASS GEN HOSp, F R U I T ST, BOSTON U2 1 1 H 

FEE- 5175 FOR- b, LIMIT 100* 

LENGTH- 5 DAYS, 3b H*S f 

METHODS- LECi LC • 0, PaN, PD • 5EM 



1 1 651 
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CARE Or THE 
SPONSOR- 



LOCATION 
FEE- *60 



entry date 05/1b/71 oelet^ date 05/20/71 
high risk mother, foetus, and newborn 
university of british columbia health science centre 
continuing education division 
Task force bldg»» Vancouver b, bc, Canada 

• same address 

• ou 



LENGTH- 3 DAYS 



METHODS- LtCTUKESi DISCISSIONS, DEMONS T* AT I ONS 
i COMMENTS- INKOKMAMUN a N D APPLICATION fDRMS AVAILABLE 

11230 ENTRY DATE. 05/19/71 DELETE DATE 05/21/71 

AMBULATORY CAKE OF CHILDREN 

SPONSOR- . UN IV OF KY COlL OF MED, DEPT oF PEDIATRICS, 

BOO ROSE ST, LEX INQTON • KEnTUc< y 40506 

FEE- *7S FOR- B , LIMIT 60 
LENGTH- J DAYS, 18 HKS, 

METHODS- AV, BR i CLjN C , LECf 0, PAN, pD, SEM, TV 

112*46 ENTRY DATE U5/23/71 DELETE DATE 05/2S/71 

\UTH ANNUAL PEDIATRIC POSTGRADUATE S YMPOS J UM-19" 1 
SPONSOR- MAIMONIDLS MEo CNTR, HB02 IOTh AVEi 

BROOKLYN i NEW YORK 112 IV 
LOCATION- CONEY ISLAND HOSP , OCEAN £, ShORE PKAYS, 
BROOKLYN I 1 2 3& 

FEE- S>75 FOR- B, LIMIT 250 

LENGTH- 3 DAYS, 2 1 HRS, 

mcthOos- aVi lEc, o, pan, sem 

1 1 2H 7 ENTRY DATE Qb/23/71 DELETE DATE 0S/2S/71 

IOTH ANNUAL PEDIATRIC POSTGRADUATE SYMPOSIUM 

SPONSOR- STATE UNIV Or NY, DOvvNSTaTe MeD C N T R , 

HSQ CLARKSON A VE, BROOKLYN, N£A YORK 11203 
LOCATION- CONEY ISLAND HOSP , OCEAN 6 Sh^RE PKWY, 
BROjjKlYw 1 1 2 35 

i FEE- S/S FOR- B, LIMIT 250 

« LENGTH- 3 DAYS* 21/ riRS, 

METHODS- AV, LLC, 0, PAN, SeM 

M20V ENTRY DATE 06/00/7 1 DELETE DATE Q6/UQ/71 

ANNUAL P L D I A T K 1 C SEMINAR 

SPONSOR- CHILDREN'S MEaLTH CNIR, 0UU1 F*OST ST, 

San d i ego i California 92123 
location- local hotels 

fee- s>30 for- b 

length- 2 1/2 days, 12 h r s • 
methods- a v , pan , s£m 

11242 entry date 0^/01/71 oelete date 06/u1/71 

pld 1 atr ics 

SPONSOR- CrilLUl.tNS MEM HOSP , S2N0 t, DEaEY AVE, 

Omaha % nE^raskA 6*105 
location- same aooress 

fee- sio for- b, limit 10. ) 

LENGTH- 1 1/2 DAYS, lU MRS • 

METHODS- A V , CLIN C, L£C, "U t PAN, PD 

COMMENTS- INFORMATION APPLICATION FORM AVAILABLE 

112MB ENTRY DATE U&/U1/71 DELET^ DATE 06/U1/71 

CLINICAL PHARMACOLOGY b PHARMACOGENETICS 
| SPONSOR- AMER ACAD OF PEDIATRICS, 1 BO 1 HlNMAN A V t t 



EVANSTON, ILL 6020^ 
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MEDICAL l.TBKARY RE QUEST FORM 



The following page U a. Library Request Form for the Middle- 
ton Medical Library of the University of Wisconsin Medical 
School. You may request & borrow a specific book or reprint or 
you may designate a subject area in which the medical librarian 
will search. This service is free of charge to all Wisconsin 
physicians. 



SPECIFIC REQUEST 

Name of Requestor ; Telephone Mo» ; 

AddreB 8 ' 

Zip Code 

Book 
Author: 

Title: 

Edition and date: 

Periodical article 
Author: 

Title: 

Journal title: 

Volume : Pages : Date : 



i 



REFERENCE REQUEST 

Haae of Requestor Telephone No* 

Address ^_ 

Zip Code 

Subject (as specific as possible): 



Aspects (Circle): Etiology, diagnosis, therapy (any kind or specifically drug 
therapy, radiotherapy or surgery), complications, statistics, other 

Subdivisions: Ag e Sex Ethnic group 

Clinical Experimental Historical 

Tims period to cover (exhaustive literature searches are no'*, done) 

Languages : 

Use (Circle): Clinical, research, paper, speech (professional group, laymen) 

Other : • 

Sources checked (i.e. Index Medicus, Current Medical References, none): 
Heeded by (date): 

Request information from: Meaical Library Service 

1305 Linden Drive 



rn 9^v Madison, Wise. 53706 



A LilBLIOGKAPHY ON MEDICAL KKCOHb K1S1SPIHG 
"A Hew Ambulatory Health Record", editoria] in Group Practi cc , 2/70. 



Baker, Terry, Nicholas J. Kiel, Peter Finklos teln , Petor 0. Ways, The 
.Problem-Oriented Hecord (A Self-Instructional Unit); College of Human 
Medicine, Michigan State University, Mast .Lansing, Michigan. 

lijorn, John C. , M.U. , and Harold D. Cross, M.D. , The Pro blem- Oriented 
Private Practice of Medicine - A System for Comprehensive Health Care., 
Modern Hospital Press , Chicago, Illinois. McGraw-Hill Publications 
Company - 1970. 

Engel, George L. , M.D. , "Care arid Feeding of the Medical Student - The 
Foundation of Professional Competence", Feb. l'j , 1971, Vol. 215, 

Ho. 7. [Keprint requests to 260 Critten* ..n 131yd., Rochester, N.Y., 
1^620, (Dr. Lngel)]. 

A Handbook for Resear ch in Gener al Practice , Liu. by T.S. Limerl, M.D., 
and A.J. Laidlaw for The Koyal College of General Practitioners, 2nd ed« , 
F. £• 3. Livingstone Ltd., Ldinburgn and London, iy69. 

Kanner, Irving P., M.D., "Programmed Medical History-Taking with or with- 
out a Computer", JAMA , 207: 317-321, 196<J. 

Kanner, Irving F. , M.D. , "The Programmed Physical Examination With or 
Without a Computer", JAMA , Feb. 22 1971 , Vol. 21b, No. 8. (Keprint re- 
quests to Dr. Kanner at 800 Rose St., Lexington, Ky. H0 l j06) . 

Lusted, Lee B. , M. D. , " Decision-Making Studies in Patient Management", 
The Hew ling land Journal of Medicine , Feb. 2 c j, 1971, Vol. 284, No. 8. 
(Reprint requests to Dr. Lusted at Dept. of Radiology , university of 
Cnicago, 9^0 East r 39th Street, Chicago, 111. 60637). 

He nne r, John H., M.D. , "The Problem-Oriented Chart (Weed System)" in the 
Summary of the Workshop in Family Practice , May 21, 1970, held in Kansas 
City, Missouri, by the A. A. CP. Commission on Education and the Sears- 
Roebuck Foundation . 
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Stein, Paul 3. , K:D. , "The Use of the Problem Oriented he cord for Teach- 
ing and Evaluation", from the Proceedings of the 1st Banff Workshop for 
Teacher:; of Family Kcdieino, 28 June to 1st JuJy 1970; Sponsored by 
Tnc Mociety of Teachers of Family Medicine and The Division of Continuing 
Medical Education , Faculty of Modi cine, The University of Calvary. 

"Ten i-.casons Why Lawrence Weed is hlpjht 1 ' , editorial in The N ew England 
Journal of Medicine , Jan. 7, 1971, Vol. "2 81, ho. 1. 

A'ecci, Lav:rence L. , "What Physicians Worry About: How to Organ- 

ize the Care of Multiple-Problem Patients", Mode rn Hospital 110: 9^-9^, 
19C8. 

Weed, Lawrence L. , M.D., "Medical hecords that Guide and Teach:, The New 
England Jo u rnal of Medicine , March 1^, 1968, Vol. 278 tio. 11; Conclusion 
in i-J.E. J.M. , March 21, 1268, Vol. 278 No. 12. 

Wee d , Law ren ce L. , M . D . , Medica l Ue cords, Medica l Educa tion, a nd Patient 
Car e, Tne Problem-Or ient ed Reco rd as a :iasl c Tool , The Press of Case 
Western heserve University, Distributed by Year Uook Medical Publishers, 
Inc., Chicago, 1970. 

Compiled by Family Practi ce Program. Post-Graduate Medicine. 



PHYSICIAN 1 S PROFILE INTERVIEWS 
TASKS DELEGATED TO ASSISTANTS BY PHYSICIAN 



1. 
2. 

ft. 

6. 
7. 



Te lephone 
Triage Screening 



Done by : 

1 

2 



M.b. 



R.N. Tech, 



Nurse- 1 s 
Asst. 



Sec. Other 



9- 
10. 
11. 
12. 
13. 
14. 

13. 
lb. 

17. 
18. 



Medical record keeping and 
retr:".val of information 

History taking 

Obtaining speciman's for 
lab worK 

Laboratory tests 

Allergy tests 

a. scratch 

b. intradermal 

he asurements* 

a. weignt & neight 

b . temperature 

c. o.P. 

ECG 1 s 

houtine pap smears 
X-rays 

Pregnancy tests 
Audiometry 
T onomet ry 
breast exams 

Pnyslcal exams (eye, ear, 
nose , throat , sometimes 
entire exaro) 



Sigmoidoscopy 

Sigmoidscopy exam assis- 
tancy 



19. Well-baby checks 

20 



3-_ 

5._ 
b._ 

Y._ 

a. 

b 

8._ 

a • 

b. ~ 

c . _ 

10 ._ 
11. _ 

13.. 
1H.. 

1';.. 

16. 
1'/. 



18. 
19. 
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, Wound dressing, application 
4 changing 20., 



21. Diagnosing under physician's 
supervision SI 



TASKS DELEGATED TO ASSISTANTS (CONT. ) 
Done by 



M. D. 



fi.N. 



Nurse ' s 
Tech. Asst. Sec. Other 



•M. 
21. 
25. 
26. 
27. 
28. 

29. 
30. 

31. 
32. 

33. 

Jl. 

3b- 
36. 
37. 

38. 

39. 

10. 

11. 
12. 

13. 
11. 



Suturing 

Suture removal 

injections 

Cast application 

Cast removal 

Physlotneraphy 



22-. 
2 3.. 
21. 

25-. 
26. 
2 7. 



Initiate emergency treatment 
for shock, cardiac arrest, etc2 8. 

Ear irrigation 29. 

blaacler irrigation & dilation 30, 

Postural drainage 31, 

Ankle taping 32. 

Surgery assistancy (other than 

W.D. ) 33. 

Patient instruction 3l< 

Patient counseling 35. 

Phoning in prescrlptl ens 3b. 

Keeping records of pre- 
scriptions 37. 

Prescribing under physician's 

supervision 38. 

Home care visits (following 

patient's progress) 39. 

Emergency house calls (when 

physician not available) 10. 

riospital rounds 11. 

Nursing home visits 12. 

Training other assistants 13. 

Maintaining patient traffic 

flew HI, 



















, 
















1 











• 



































































































- 









































- 








. — — 
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EDUCATIONAL RESOURCE I N DEX REQUEST FOHM 



DISE ASE CATEGORIES Audio- Prog. Texts & 

Conf. P..T. Visual Inst. Journals 



1-1. Infective & Parasitic 



2-1. Malignant neoplasms not cf blood 












2-2. Neoplasms of Blood & Lymphatic 












2-3» Benign neoplasms 












Neoplasms unspec. 






— 






3-1. Diabetes 












3-2. Obesity 












3-3* Enao. , Hetao. & Nutritional 












4-1. Diseases of the Blood 












3-1. Mental Disorders 












6^1. Diseases of the C.N.S. 












6-2. Eye & Ear 












7-1. hneumatic Fever & Heart Disease 












7-2. hypertension 












7~3» Ischemic heart disease 






H 






7-^. Other heart diseases 


s 








7-5. Cerebrovascular disease 


1 
1 








7-6. Other Vascular Diseases 












7-7. Cardiovascular unspecified 












8-1. Acute Respiratory Infections 












8-2. broncnitis , Emphysema, Astnma 












Otnex 1 hespircitory Diseases 












8-4. Respiratory System unspec. 












9-1. Oral cavity, salivary planus, jaws 












9 . Esophagus, stomach & duodenum 












Intestine & peritoneum 












9-4. Liver, gallbladder, ic pancreas 












9-9. Gastrointestinal System unspec. 












10-1. Urinary System 












13-2. Male genital organs 












10-3» Breast u Female genital organs 













10-4. Genitourinary System unspec. 












11-1. Comp . of Pregnancy & Puerperium 












11-2. Delivery 












12-1. SKin & 3 u be ut ane ous Tissue 








h 




13-1. Artnritis & Rheumatism 








! 


13-2. Mus . skeltal. sys. , Conn.. Tissue 












1^-1. Congenital Anomalies 












15-1 • Perinatal morbidity ano mortality 












lu-l. Symptoms & Ill-aefinea conditions 












17-1* Fract., Disk., Sprains, Strains 












17-2. Other trauma incl. burns 












17-3* Adverse Drug effects and poisoning 












17-r^. Other adverse effects 












18-1. General Exam 












18-2. Radiological Examination 












18-3. Well baby & child care 












18-'!. Pre-natal Exam 












18-5 . Post-partum 












18, m. Contraception 












lc , . Otner Exams w/out Illness 












lb-8. Post-op. care & Rehabilitation 
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Exhibit 9 

Participant No. ^ 2~S £ l» * 

I . P . P . Pol low-up 
June '71 

Utilizing t lie following scale in questions 1, II and III check the box 
that oost describes your opinion. 

L. As to recognizing my educational needs, i found participation in 
I -P.P. to dc : 

Very Helpful Helpful ,Not Helpful 

Q EI H 0^0 0 EI 12 

11. As to the procedures used in I. P. P. I found: 

Very Helpful Helpful Not Helpful 

A. Profiling 

(pt. problems) ^/ 

□ El 0 

B. examination ^ 

CD 0 0 

t. Consultation 

ID ta El 

0. Ihe whole program ^/ 

a m oi 

III. 1 found the information about my practice {% females, °» of patients 
under 4U, av . number of office pts seen/day etc.) 

Very interesting interesting of no interest 

y 

a a a 0ia 0013 

In questions IV through VII please make the indicated response. 

IV. Have you completed one or more of the recommended educational pro- 
grams? y 

Yes * No 



01 


a 






0 B 


0 




a 


J3 




13 


a 


B 






0 


0 


0 


Gl 


a a 



A. If yes, wnicn one or ones (please describe) 
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B. If yes, did it or they Meet or fail to meet your needs? 
(please comment) 



Did you find the print out from the Educational Resource Index 
(recommended educational programsj easy to understand and use? 

Yes X/^ No 

A. if you answered "no" please comment why. 



VI. Would you consent to a second (post) test? ^/^ 

Yes No 



VII. Hrom your standpoint what are the "bugs" in I. P. P.? (don't be 
kind, use the other side and an extra sheet if necessary, your 
criticisms can only be beneficial) ' 

Tb~sT sh**?T- Skcutd 6e *J>Jc 7* 
U " T ** 7**7- foists V »nc^i 



*-in thousands 



Exhibit 10 
General Prac fri ti oner s 



i 



ysic ian 
Code # 


Physician 
Age * 


Yrs • in 
Practice 


Size of 
Community* 


# of Drs. 
in Community 


10016 


36 


11 


2 


2 


13029 


47 


20 


3 ' 


3 


14056 


31 


5 


163 


4 75 


1502B 


40 


13 


37 


69 


16049 


41 


14 


5 


"X 


16218 


45 


2 3 




1 i 


16347 


40 


15 


7 


0 


20021 


40 


11 


163 


475 


20045 


31 


5 


7 


Q 

j 


20080 


38 


13 


10 


x x 


21455 


42 


18 


3 


3 


22222 


57 


32 


34 


22 


23663 


3 1 


c 
O 


1 A 

1 4 


15 


24093 


53 


24 


8 


6 


24246 


49 


25 


34 


60 


25045 


30 


5 


2 


4 


25573 


34 


8 


6 


1 1 


26044 


55 


29 


7 


10 

X V 


26225 


45 


21 


2 


7 


32053 


41 


16 


7 


10 

X J 


35006 


39 


9 


16 3 


475 


35061 


59 


32 


2 


3 


36151 


41 


8 


2 


4 


36247 


37 


11 


34 


60 


36375 


41 


10 


5 


11 


42083 


65 


33 


4 


5 



Medical 



Wise . 

1959 
Wise. 

1950 

Minn . 
1965 

Cinnci. 

1957 
Nebraska 

1956 
Wise . 

1947 
Wise • 

1955 
Wise • 

1959 
Marquette 

1966 
Wisconsin 

1957 
Wisconsin 

1952 
Wisconsin 

1938 

Mi chigan 

Wii&fensin 

1946 
Wisconsin 

1945 
Wisconsin 

1965 
Minnesota 

1962 
Marquette 

1941 
Wisconsin 

1949 
Wisconsin 

1954 
Wisconsin 

1961 
Wis cons in 

1938 
Marcruette 

1962 
Georgetown 

1959 
Wisconsin 

1960 
Wisconsin 

1937 



*=*in thousands 







vjcnci d J. 


r idt l lLioncrs 






iysic ian 


Physician 


Yrs ♦ in 


Size of 


# of Drs. 


Medical 


Code # 


Age 




cvjiruiiunx in 


Co nunun i t y 


Education 












W iscons in 


42553 


46 


19 


2 


3 


1951 
Wiscons in 


43675 


40 


15 




Q 

o 


17J J 


45529 


36 


9 


8 


9 


1961 


46078 


49 


25 


8 


15 


194 5 
Manitoba 


46540 


44 


15 


4 


1 


1955 
Wiscons in 


50016 


50 


25 


4 


7 


1945 


51323 










Maryland 


40 


9 


1 


1 


1961 
Iowa 


52017 


38 


12 


13 


31 


1958 
Illinois 


52030 


41 


16 


2 


1 


1954 
Wisconsin 


52047 


33 


6 


5 


3 


1964 
111 inois 


52619 


33 


8 


8 


9 


1962 
Indiana 


54053 


35 


10 


3 


8 


19 60 
Iowa 


55074 


41 


13 


13 


31 


1957 
Wisconsin 


55076 


36 


6 


5 


4 


1964 
Wiscons in 


55232 


31 


6 


2 


4 


1964 

INeJjx dSKd 


55247 


33 


8 


7 


10 


1962 
Wiscons in 


55352 


39 


8 


38 


D V 


lo i unto us f w « 


55649 


30 


5 


2 


4 


1965 
ina ldua 


56343 


36 


10 


1 


3 


I960 
C inncinria ti 


60045 


44 


19 


163 


475 


1951 

AllKS Vlllcf 


60053 


40 


15 


741 


1105 


1956 


60089 












48 


21 


3 


5 

-J 


1 Q47 

Iowa 


61275 


42 


17 


163 


475 


1953 
Wisconsin 


61417 


48 


24 


6 


11 


1946 
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Exhibit 11 



Physician/Patient Data 

Total number of physician's - 76 
63 G.P, 
7 Peds 
6 I.M. 
1 G.S. 

Total # of Diagnosis in 18 Categories 
by lst-6th diagnosis classification 



Category 


1st 


2nd 


3rd 


4th 


5th 


6 th 


Total 


± 


J H H 




37 


1 2 


3 


1 


732 


2 


370 


214 


c o 
DO 


2 o 


9 


j 


COO 


3 


558 


342 


152 


50 


18 


5 


1,133 


4 


89 


99 


44* 


12 


7 




251 


5 


492 


242 


90 


27 


13 




864 


6 


606 


215 


78 


28 


8 


4 


9 39 


7 


1,234 


794 


336 


140 


42 


18 


2,564 


8 


2,043 


656 


162 


38 


15 


3 


2,917 


9 


670 


299 


118 


37 


21 


9 


1,154 


10 


683 


286 


110 


36 


19 


9 


1,143 


11 


96 


67 


6 


2 






171 


12 


485 


195 


51 


19 


■ 6 


3 


759 


13 


448 


179 


69 


21 


9 


7 


733 


14 


76 


58 


24 


8 


1 


1 


168 


15 


50 


16" 


5 


3 






74 


16 


1,234 


760 


286 


101 


25 


3 


2,409 


17 


1,569 


412 


104 


31 


19 


4 


2,139 


18 


3,239 ' 


1,346 


353 


81 


29 


21 


5,069 


Total 


14,486 - 


6,315 


2,093 


682 


244 


91 


23,911 




Infective and Parasitic 
Neoplasms 

Endo. , Nutrition, Metab. 
Blood, Blood-Forming Organs 
Mental Disorders 
Nervous System, Sense Organs 
Circulatory System 
Respiratory System 
Digestive System 



10. 
11. 
12. 
13. 
14. 

15. 
16. 

17. 
18. 



Genitourinary System 
Preg. Childbirth, Puerperium 
Skin and Subcutaneous Tissue 
Mus • s ke It al . Conn • Tis s ue 
Congenital Anomalies 
Perinatal Morbidity, Mortal* 
Symptoms, Ill-Defined Cond. 
Pract*, Trauma, Poisoning 
Spec. Cond,, Exam iSi^to,e§.3^ 



Appendix A 
Development of a Clinical Test Bank 



Testing of physicians to determine the level of their medical knowledge, 
under the best of conditions, is a difficult undertaking. The Individual Physician 
Profile project required such testing as a major part of the process; it also 
imposed additional challenges: (1) there must be the capability of testing in any 
area of clinical medicine, (2) each test instrument must be developed in relation 
to a specific medical practice, and (3) this testing capability must be developed 
under severe time constraints. 

It was decided that the project required a comprehensive test bank from 
which appropriate questions could be selected to devise the individual test 
instruments required. Since each test would be different, it was not considered 
possible to utilize traditional methods of establishing the validity and reliability 
of each instrument. Consequently, an alternate approach of devising a rationale 
and procedure which, if followed, could be expected to result in a degree of 
validity and reliability was developed. 

This approach concentrates on establishing the quality of each question 
entered into the test bank, with the assumption that a test instrument composed 
of such questions on the basis of certain criteria retains that element of quality. 
It makes further assumptions that this quality can be established on, the basis of 
review by clinicians and subject area specialists. 

In order to obtain the quality judgments of clinicians and subject area 
specialists, an item review cycle was established; this is diagramed or. the 
following page. 

The main course of an item through the review process is: (1) acquisition 
either from an outside source or written by a staff physician, (2) clinical review 
by mailing out to a practicing physician and obtaining his judgment on a rating 
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ITEM REVIEW CYCLE 



REWRITE 



ACQUISITION OF ITEM 



ITEM WRITER 



OUTSIDE SOURCE 



STAFF REVIEW 



V 



> | CLINICAL REVIEW 1 



STAFF REVIEW 



> STAFF EDITING 



SUBJECT AREA 
SPECIALIST REVIEW 



STAFF REVIEW 



TEST BANK 



sheet (Exhibit 1), and (3) scientific review - by mailing to the subject area 
specialist on the faculty and obtaining his judgment on a rating sheet (Exhibit 2). 
At each point along the way there is staff review with a judgment as to whether 
the item can proceed to the next step, requires minor revision, requires major 
revision, or should be rejected. In addition, once an item passes clinical review, 
it undergoes editing to assure proper format, spelling, etc. 

Experience with the cycling process gave a certain degree of confidence 
that each item was subjected to critical review by the clinicians and subject area 
specialists. There was a high attrition rate, even though the majority of the 
items had been used in one form or another for testing physicians or those in 
medical training. The majority of items that survived the cycle were diverted 
from the main course at one or more points in the cycle for re -writing. 
Consequently, it has been concluded that the physicians involved in the cycling 
process performed their assigned tasks and, if the criteria used are valid, the 
test bank constitutes a unique resource in testing on various aspects of clinical 
medicine. 

With the knowledge that a substantial test bank must he developed in a 
relatively short time, a number of experiments were conducted with the intent 
of assuring proper quality judgments on the part of the clinicians involved, and 
determining the most efficient methods by which this could be done. The 
experiments tested the following hypotheses: 

Hypothesis M . Individual clinicians making value judgments on the items 
do not represent a significant source of variation. 

In the interests of economy of time, it was proposed that each item be 
cycled through only one clinician; this required that a degree of inter- rater 
reliability be established. In the experiments conducted, individual items were 
cycled through a number of physicians and their ratings compared. 

In general, the factors related to clinician judges tended to show no 
significant source of variation. 

Hypothesis -2. Staff judgments on the general quality of an item will agree 
with the responses of the clinician judges. 

As non- medical staff carried out their duties in cycling items, they developed 
; an ability to make judgments as to whether a specific item was "good" or "bad", 
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or of "questionable" value. The experiments confirmed that the judgments of 
"good" and "bad" questions correlated with those of the clinician judges. The 
clinicians, however, rated the "questionable" items more favorably than either 
of the extremes. 

The result was that the staff could be more efficient by immediately 
rejecting those items they judged "bad", and not taking up the valuable time of 
the clinician in confirming this judgment. Therefore, only those questions that 
could be expected to survive the review process would be cycled. The results 
gave equal justification for by-passing clinical review of those items judged 
"good" by the staff, but suggestions made by clinicians for improving these 
questions by re-writing were considered sufficiently valuable to v/arrant the time 
and effort involved in cycling them. 

Hypothesis £3 . Non- medical staff members are able to improve the quality 
of items by revising them prior to cycling. 

Non- medical staff members expressed confidence that they could take items 
initially judged as "bad", and by re-writing them improve the quality sufficiently 
to make them acceptable for inclusion in the test bank. Since previous experi- 
ments had established correlation between the staff judgments of "good" and 
"bad" items and those of the clinician judges, a number of the "bad" items were 
re-written by non-medical staff and cycled along with a control group of items 
which had initially been judgod "good" by the staff. 

The results of the experiment indicated that the clinician judges gave the 
same ratings to the modified items as to those initially judged "good". Conse- 
quently, attrition of items could be reduced by having non- medical staff members 
revise certain ores. 

Hypothesis #4 , The fact that some judges are given the opportunity to opt 
out of responding to an item does not affect the pattern of responses. 

The clinician judges were asked to make their judgments in the broad terms 
of clinical practice, rather than in reference to their specific practices. It was 
thought that some reviewers might feel unqualified, or at least uncomfortable, 
when an item dealt with a topic where they had no direct experience. For this 
reason, specific items were cycled to two groups of physicians; one was required 
to make judgments on all items, and the other was given the option of not 
responding to certain items, 
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The results of the experiment indicated that it made no difference in the 
item ratings if a judge had the ability to opt out, or not. Consequently, the 
necessity to re-cycle certain questions where the initial judge did not respond 
was avoided. 

Hypothesis #5 . The method used to print an item does not affect the 
response given to the item. 

The most convenient method by which the staff could transmit a specific 
item to a judge was in computer printout. Since this was in capital letters, and 
in a form unfamiliar to the clinician, it was thought this might have an effect 
on the results. Consequently, the same items were cycled to clinician judges in 
computer printout and typewritten format. 

The method used to print the item did not noticeably affect the response 
given to the item, and consequently the more convenient computer printout form 
could be used. 

Hypothesis £6 . Including a reminder concerning the criticality of careful 
responses dees not affect the pattern of responses. 

It was feared that as a clinician became more familiar with the review 
process, his consideration of individual items would become more hurried and 
superficial. Consequently, in the final experiment the same items were sent to 
two groups; one recieved a reminder of the critical nature of the judgments and 
the other did not. 

Results of the experiment indicated that the reminder did not affect the 
rating given an item, implying that the clinicians did not need to be reminded of 
the importance of their judgments. 

Hypothesis ?7 » The number of items a clinician is asked to review at one 
time may affect his efficiency and effectiveness. 

There was need to cycle the maximum number of questions in the shortest 
possible time, consequently data were desirable on the most reasonable number 
to send. One factor was that a busy physician might be more likely to promptly 
complete review of a limited number of questions, but would put aside a greater 
number until he had a sufficient block of time available to complete the task. 
A second factor involved the possibility of fatigue, which would make judgments 
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on the last questions reviewed less valuable. Consequently, packets containing 
varying numbers of questions were sent to different groups of physicians. 

The results indicated thai physicians were more likely to return packets 
containing 12 items promptly than those containing a greater number of items. 
The ratings involving packets containing J 2 questions also showed less variance 
than those containing more items. 

Since the item cycle and experiments may be of value to others interested 
in medical testing, they will be reported in greater detail at a later date, including 
experimental design, data obtained, statistical treatment of the data, and basis 
for the conclusions. 

As a result of the cycling process, and the experiments conducted prior 
to development of the test Dank, the study staff has confidence that a unique 
resource has been developed to meet the requirements of the research. Equally 
important, the test bank of approximately 2,000 items is now considered the 
base of a more substantial test bank which, with additions and modifications, 
can serve a variety of needs in testing throughout the continuum of medical 
education. 
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IMF A 7 T A f H F "I MUSCLE PULL 

TM°R op r r spltnttng * 
iarp7n 2oon 7P n 

TW^ n A Y S A^O A «iO vfap OLH MQIJSF.WTFF I NJURFD HF °5>ELF WTTH A RAD HEN 
STCKF INDUCING A L fi PFR A TT o N OF TMr SKIN IN T MF SIGHT PPPL TTFAL SPflFF,. 
AT TriF PPFSFNT T TM r « THF WOUNO. AN 0 SURROUNDING TIS^UF APPFAP. INFFCTFD + 
SHP S T f . T F c THAT SH^ HAD NpVFR R ^F N IMMUNT7FP A^AIN^T TETANUS, ft PT* P 

clfa^stnh, nfqrjtqfmfnt and ppfssing of thf wounp. thf p f s t trfatmfmt 
woulh rf: 

•^INTVFNT Afsjn ALUM P C I * I T A TFl> . T r T A NU S TO X 0T n 
A NO AQUFOUS T F T A NLJ S TOXpTD 

OINTMFNT ANH ^tOPfJ UNITS OF F gg T N.F _ T _F_ T A MJ S ANTITOXIN 

A NT UOO UNTT^ OF HUMAN TFT A NU S a'NTITOXlM 

A NO LOCAL ^APITPACIN CTN T M F"NT « 

1 HO 1 7 P F 

F^ACTU^FS OF TH^ F F MOP A L SH^FT TN CHTL HRFM « >PYANT»S TR*i TTT ON 7^ 
^TT)rorp THF TRFATMFNT ^ THOTCF UP TO WHAT A^rt 



RACIT7ACTM 
PFN'TCILLTN 
RAT IT I? ATTN 
PFNTC ILL TN 
P^NI^TLLIN 



TN 
CO 
A. 
P. 

r« 
n. 

F « 

C C1' 



ERIC 



TWO Y r ARS 
FOUR YFAPS 
SIX Y^APS 
T F N YFARS 
ROTH APF ANP 
1 70RGP 

WHIPH Ofi r OF T"- FOLLOWING 
TO THF AN T F p I 0^ CHFSf: 
A • n TS S FC T T 0 N OF THF AORT A 
CONTUSION OF THF LUN'n^ 



WFTGHT SHOULD D F COWSTPFPFP 

3 run 

IS NOT A POSSIRLF 



7P 



CnNSFf>|JFNCP OF TRA IJMA 



n at 



ERIC 



T77^ns c. ^apaooxtcal fmrolization 

377<*OG n. H r MO n t>H&ntC °F R T C A&0 T T TS 

T7( 17 r. PRFWflTUPF V r VTPTOUL A° CONTRACTIONS . * 

3:>97 17^14 17TF75 17RS13 17A807 ?PDl F9 A 

33P7U1 WHICH of tmf following p-qattupfs which may not pf tnttully fvtpfnt 

37*70? PAOTOnPAPHTrALLY MAY RTV^ PISf TO.. CHRONIC C LIN LC_A L Q I SAB LL I IY_ I - 

SS^O"* 1 A. PPAfTu^^ OP THF C APPAL NAVTOULAR 

339704 R • MA9CH F5ACTU*F 

3 7 97p* C. PT^ FoftfTUPT 

33970*) n, POMMTNUTFO. COLLFS • roA^TUPF 

3397P7 r . rpAfTU^ 1 " OF TMF «ASF OF T HF ST H MFTftTARSAL $ 

29*4 17S<^5 1*P7<}£ 1 njl 1 _ „ 7.0 f).. 

" 2O3401 & YOUN^" MAM T S RPOUHHT 'INTO THF F M F R GF N r Y ROOM WITH ACllTF PP"SPTRATO D Y 

2QR407 FA T LU°F HUF TO A HF A D INJURY. YOUR FMFPGENCY OARF WOULD INCLUPF ALL 

?9B<i03 THF FOLLOWING F XC^PT W^rrH 0 NF ! 

?*BU04 A. FSTARtTSH 5TPWBY 

2 Q 34PS P. GIVF PFSUSCTT ATTVF V r N TT L AT T ON 

29?anF C. r T V r OVYGFN . „ 

7^8407 n. PTVF SFRATTVF FOR PftTN 

?9°.un« r. f x & ^ t fc j f to fxcludf r n tf^ n al q lff oi ng s 

?p.i i i 709t* ^ 2oi i 7p n 

73 1 1 HI TN SPTTF Or ADVANCES IN TRFATMFNTi THF MOST COMMON CAU^F O r DFATH IA' 

2*11 07 RUPM CHS*" S TS ! 

7 P . VI 03 A • *UPN SHOCK _ . „_ 

~?8llGu fc. MFPATTf COMA 

?8110 c C, HCMOPPHAPF r ROM A HjPL IN G • S ULCFQ 

2°lllln H. TN c FTTTON 

2X/*(M F. FLFCTPOLYTF TM&ALANCF % 

I i\ i 70 * r* o 3110 c 9 n 

1176G1 WHTCH OF THE. F OL LOUT N^ 4»F J HF MOST FASTLY OFHCNS.TPABLF F ARL.Y Jftfl.M-T 

11250? rFS TAT TON^ 0 r *ONF MAPPOW TN H IB T T T ON FPOM 0 MLO P ft MP HF N T COL 

117fctn (CMLOPOMYCFT IN ?X ) ! 

117SU4 a. NUCLFATFO P^O RLOOP C F LL S A NO MTTOTTC FTGU° r S TN THF" RLOOD 

U2G0F, P. THPOM^ory TOPFNT 6 ANO OFCPFASFO STICKINFSS OF PLflTFLFTS 

117^0^ C. *FC- ALO^LAST TC CHANG** TN THF PONF MARROW ANO Mft CPO C Y TF S TN T U F 

1J7CP7 PFP I P H F A L PLOOT _ _ 

117P.0P 0- ? C T TCULOC YTOPFNT A MY PF P FF D P* 7 MT A 

II ?S0^ F- °ONF MARROW A^L fi S T A ANO P A NCY T OP FN I A « 

l o m o i f i ? ? no 1 p; p q 

10400! k'HTr H OF THF FOLLOWING FPAC T UPFS TS USUALLY TP r a TFO WITHOUT PL A S TF P 

lo^inn? TM^n^TLT^aTTON: 

104003 &« T^ANSVFFS^ PfiTFLLAP _ 

"Tri^nOU P. TMPACTFD SUPrTCfiL N^CK OF HUMFRUS 

lU4Qn c C. GRFtN^TITK i-PACTURE OF °0 T H ^ONFS OF FORFAR M 

ipunn^; n. mto-ma vtcul* * 

lOUnP? F « M F 0 T A L MALLFOLUS OF TTBIA s 

107T 170°01 3077 co ^ 

jn?701 SFVrnr RU p N s pgpOUCF WHAT FFFFCT UPON T ME LF VF L OF GAMilfl fiOffillTM T K' 

107707 THF FIRST W F F K FCLLCWTN^ THF PUPNt 

10770^ A « ^APKFT TNCPFASF 

107704 k. MAPKFO ^rC? r ASF 

107705 C. NO S T^NiTF ICAMT OHAN!=F S 
10 1 7 I 7F P S 4 I DO 1 P q C 
Iff 11 a ^^'JNfi MAN IS 6 DM T J TEO TO THF HOSPITAL, FOLLOW TNG A FALL FROM A HOPSF 

"Y0r>0^ IN WHICH HF STRUCK HIS HFflO UPON A FFNCF POLF. ALTHOUGH HIS SFN^ORTUM 

101703 WAS NORMAL UPON A DM T SS T ON t WlTMI^ ONE HOUR HF TS STUPOROUS* THF PTfTHT 

?JPP«i PUPTl T S OILATFO ANO T H r L r F T CONSTR TCTFD • THF FMFPGFNCY TPF AT MF NT 
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Appendix D 



Medical Questional re System 



T n t r orluc t i on 

The Medical Questionairc System is a qroup of A pmarnrrs called 
Input, PPC, TTRA , QOUT. It was desiqned to produce individual ir'.od 
examinations for physicians based upon the diaanoses each phvsician 
made on h 1 s pa tients , 



Hencral Approach 

All Medical questions and nhysician/patient data are coded into 
tile ICDA (revised) classification of diseases and also accordina to 
speciality levels. The patient data and physician speciality is 
read by proqram ^'J 5 -^ The medical questions information is road 
by proaram Innirbr ~ 

Proaram TTRA - loads the random access files 7 & ? 

Proaram Oout - aenerates a medical examination based on the Practice 
prof i le . 



Physician 
(special ity) 



Patient 
(Diaqnosis ) 



.Data 



coded into ICDA 
classi fi cat ion 



TEST RANK 
Medical questions coded 
into ICDA classification and 
speciality and levels of 
dif f icultv 



Medical questional. re 
System - 



Pronram "PPC" 
counts # of Dx in 
S.S. cateaorics 



Practice Profile 
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Proaram "Innut" 
(maintains Test Data File) 
(sorts auestions ^'s into 
S.S. cateaorics ) 



Proaram "TTRA" 
Loads Pandom Fi les 
7 - Innuts Tables 
H - (TEST BANK) 



Proaram "Oout" 
uses TEST formula to search 
question tables for questions 
related to Practice Profile 



. Individualized TEST 



-2- 



Lanquaqe 

The proqram is written in Fortran V and Assembler lanauaae to run 
on the Univac 1108 usinq a medium density , 7 tract odd paritv tape. 
The system uses some of the features of the 110R operatina system. 

Limitat ions of Proqram 

1. A maximum of 12,000 questions in Test Bank. 

2. Questions can be coded into a maximum of BOO R.R. categories. 
Our ICDA classification has 912. 

3. A maximum of 150 questions can be coded into one S.S. cateqory. 

4. A question has a maximum lenqth of 100 cards. 



Or q a nization of th e Program 

The system has two basic operations: 



1. File Maintenance - necessary to change or add information to 

the file. This updating is done by oroqram Input . The un- 
date cards are made out according to standard rules for up- 
dating Univac 110S files, usincf the correction statements* 



2 % File retrieval - the main function of the retrieval operation 
is to produce an examination of medical guestions based 
upon the ohysician oatient d?.ta (practice nrofile) 



General Considerations 

The information on the question cards and the physician patient 
data cards must conform to a definite format to function v/ithin the 
program. 



A series of safe guards is built into the proqram to catch errors 
in the input data. Discovery of an error will suopress execution o r 
the data set , but will allow the input to be read and nrinted and later 
checked for errors. 



Us i nq t he P roq r am 

1 n Put P rogram 

This program undat-es the test data file, (file 2 of Master Tane TO) 
performs error checking and sets up the necessary tables to search the 
test bank. It creates the new master taoe TO. 

"INPUT" uses a segmented load routine. " Inp ut 11 (main) driver calls 
r outines INI, IN2, IN3 . The scratch file units "used by Input and what 
is stored on them is as follows; 



8-25 



(ono for each category (1-18) - ques tion soanonco numbers i n 
catcaories . Number of S . S . categories are noted ) 



26- (number; of questions in S.S. catocfori.es number of questions 

in levels of speciality in S.S. catogor i es) 

27- (accession number for question, level number of question, 

starting card number for question , number of cards in each 
question) 

23- (question sequence numbers are stored here after beinrr sorted 
by S - S . categories) 

29- Master Tape (TQ) 



Input (main) - Driver 
Routines 



Pass I 



Pass II 



Pass III 



INi (main subroutine) 
Reads 

Checks coding errors 
Creates intermediate 
Piles 

8-25 (Category 1-18) 



IN 2 (main subroutine ) 
sorts question * into 
S . S . categories 



Tins - reads Files 8-25j 



IN3 - (main subroutine) 
Writes out 
TO (new Master File) 



Tidio - reads File 27 



Tout - writes out 
Files R-25 



of 



categories 



0 # in S.S. category 



TIDOO - writes Pile 27 
Acc. # of question 
Level # of question 

# of cards in question 



I Tin links Tins + IN2 i 



Sort 2 - Schell sort 



■ TOTO - Writes File 28 

sorted question # ' s 
j ' in sequence 



"1 



INPO - reads Fil<* 28 j 



TPMDO - writes File 20 
(new Master File) 



! TFSO - TPNO - Block incr 
j for IN 3 (Tables for 
' retrieval nroaram) 



-4- 



P.P.C._ Prog ram 

This program reads Physician specicility codo and coded diagnoses in 
card imaqes, (one* card per patient, six possible diaanoses/patient ) 

The proqram counts and prints out the total number of diaanoses in 
each sub-sub category resulting in the practice Profile. 

Other calculations which are made, but not printed are: 

1. Number of S.S. categories with diaanoses. 

2. Number of S.S. categories in each category with diaanoses. 
3.. Number of total diagnoses in each category. 



Physician Patient 
Data 

(coded by speciality + diaanoses) 



+ 

: "p.p.c 1 ! 

J program ' 



j Physician j 
I Profile J 



TTRA Pro gram 

This program reads in "TO" (Master File) and writes or loads units 
"7 8" in random access format. 

"Unit 7" contains the test Data File (File 2 of Master Tape "TO"). 
It is a" series of question information tables which point to Questions 
in the Question Bank. 

"Unit^jV contains the Question Bank. 

These files are to be used by "Qout" proaram to generate the examina- 
tion. 
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Appendix E 



Educational Resource Index 
"Rats " System 

Introduction 

"Rats" is an acronym for an information Jjetrleval system using 
an Associated Tree Structure format. It uses th^ hierarchical file 
structure of levels. Postgraduate educational .tiaterials for the 
health sciences professions can be categorized into four levels. 
We are using the program to provide physicians with a list of edu- 
cational opportunities in specific areas of Medicine , to suoplement 
a project entitled "Individual Physician Profile." 



General Approach 

All educational materials are coded into a modified version of 
the ICDA (revised) consisting of 54 categories (see Appendix - Ex- 
hibit 1). The information is retrieved by listing level names. 
The following flow chart will illustrate the structure and retrieval 
aspects of the program. 



ERLC 



2. 



RATS 



¥ 

DOCTOR 



Level 1 (Profession) 



V 



INFECTIVE AND 
PARASITIC DISEASES 



DIABETES 



Level 2 (5 4 Categories) 





Level 3 (Place) 



POST-GRADUATE 
TRAINEESHIPS 



AUDIO 
VISUALS 



V 



PROGRAMMED 




INSTRUCTION 


(Type 





-Level 4 



MEDICAL TEXTS 
& JOURNALS 



INFECTIOUS DISEASES 

Sponsor - American Academy of Pediatrics 
P.O. Box 1034 
Evanston, Illinois 60284 
Location - Tulane University of Medicine 
New Orleans , Louisiana 
Fee - $75* ~ members $100. - non-members 

Length 3 days, 24 hours 
Method - Lecture, Panel Discussion 



Level 5 
Individual Entries 



[Retrieval listing level names] 
^ [ 



*Get, Doctor, Infective fit Parasitic Diseases, Away, Conferences. 
(Level 1) (Level 2) (Level 3) (Level 4) 



| following the arrows in the flow chart, the above retrieval card will list 
out the sample resource material. 

ERJC 



3. 



Thus, usinq this structured system, only material useful to 
a physician is utilised. For example: 

A physician who states he cannot leave hip practice will not 
be sent "Away" materials, but "Home 11 resources will servo his pur- 
pose. If he states he learns "by doing," he will be quided into 
postgraduate traineeships . It is a tailor-made program for the 
area of Medicine and the type of resource suitable for the indi- 
vidual physician . 



Added Feature 

An added capability of the system makes it serve more as a 
net rather than like a tree structure. If the first cateaory had 
the entries designated for doctors and nurses, entries do not have 
to be duplicated in both professional fields. 

They can be connected by nodes within the levels or unward or 
downward between different levels. 



Language 

The program is written in Cobal, to run on the Univac 1108. 



Organization of the Program 

The "Rats" system has three basic operations : (1) File Cre- 
ation, (2) Retrieval and (1) File Maintenance. 

1. File Creation 

This operation is done only once for a particular set of 
data simply because no previous file exists. The data is 
supplied by the user. The file, thus created is used by 
the Retrieval system to retrieve materials requested by the 
user. Creation is accomplished by proper set of systems 
control cards and proaram control cards (Exhibit 2 and 2a) . 
Durinq the create run a table of possible names is provided 
for all level names. A number of possible levels is de- 
termined at that time. Various control cards require a 
list of these names, one from each level. This list is 
called a classification and in each classification a name 
for each level is necessary. The order of the names de- 
termines the order these names will appear in the retrieval 
print-outs. 

2 . Retrieval 

The main function for the retrieval program is *qet. It 
is followed by a classification and is used in many ways. 

Two added functions will allow extra information to be 
printed in the retrieval r>rint-out„ 
O *HDO - prints out any information desired by user, then 

eric — 



ejects a paqe first and prints-out on the top of 
the next paqe . 

CMT - (Comment) - prints the rest of the card as it is. 

These functions can be interspersed in the retrieval requests 
and identify different sea.nents of the print-out. 



3 . File Maintenance 

It becomes necessary to 
because of mistakes in the 
formation. Updatina falls 



chancre information on a .f ile 
creation or due to obsolete in- 
into three functions , 



*Add - adds a new entry. 

*Update - chancres one or more items in an entry. 
*Also - links nodes within the system. 



General Considerations 

Each entry is a aroun of cards on which anythinq may be printed. 
This information is referred to as a blurb and is printed upon re- 
trieval and is not used for control information. Each entry , which 
consists of one or more cards is delimited by an *ADD control state- 
ment. This enables the user to submit entries in either the create 
or update node. 



Control Cards 

All control cards have a basic format. Each one has an asterick(*) 
in column 1 followed by a function name. All items which follow are 
peculiar to each function name and are separated by commas in the 
free field format. The function names used are *GET, *CMT, *HDG , 
*ADD, *DROP, * UP DATE , *ALSO. 



Using the Program 

The !, Rats n system consists of eight procrrams and three sorts. 
Communication between these programs is accomplished throuah storaqe 
on Disk. 

Create System 
(uses all programs used in update system) 

1 • Make name 

This program creates two files called "nametable" and "names . " 



1 \ d by print, 

Program ^ 

trieve 



EMC 



Input 

(List of names Possible for each level) 



I 



File "Names" 
(Card imaae of input) 



File "Name table" 
Symbol Table of Names) 



-MJsed by Edit, 
+• ask proqram 
(Saved on tape 

as Master File 



5. 



update system 



2. Edit 



This program reads update control cards and transforms thorn into 
control records by the update program. 



Update 
Control 
Cards 



Tree Also 
Control 
Cards 



ID Also 
Control 
Cards 



v 

| File' 1 Raw Data" 
I. 



V 



'Also tree 1 
File , 



% *ID Also" 
File 



Name Tables 
+ 

Raw Data 

\S + n 

Also Tree 
+ 



ID fllso 



^> File "update" 



3 * Sort Updates 

This proqram sorts the "updates" file. CJpdatincr is done se- 
quentially; therefore, the update control information must he in 
sequential order. Thp Functions for a single ID always must apnear 
in a fixed order: DROP , ADD , UPDATE , ALSO. 



4 . Updates 



"^Master File" 
^Update File" 



-~> S . Master 



5 . Sortmaster 

This program sorts the "S. Master" File bv catenories nrenarina 
it for input to make tree. 
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6 . Make Tree 



This program uses S. Master and produces two files 



fi. 



VX S. Master" ?ile 



used by retrieval 
program (must be 
recreated by each 
update) . 



* Tree" 




" Ran" 


File 




File 



i 



Random access version 
of S . Master 



6A. Link 



jfllso'-Tree File 
+ 

,1 Tree" File 



new Tree file 



Note: tree-also' control cards must be included in each update 
run unless no chanqes are made or if*also tree* file is 
saved on. tape . 



Retrieval System 



7 . Ask 



Ask reads the retrieval control cards and translates them into 



" Raw qets " 

Load qets 
+ 

Nametable 



for File "Request 


ii 


Retrieval Control -Cards 


1 


t 



File 
Raw Gets' 



File 
Load qets 



File' Request' 



Retrieve 

This proqram uses 

Files 



ERIC 




File 

' if 

Request 



-> 



Listing of re-j 
quests of user i 



